-
Best Practices in Confluence Administration – Attachments
Introduction
Attachments are a special class citizen in Atlassian Confluence but a lot of Confluence admins do not pay attention on them until bigger problem starts to surface.
This comprehensive guide comes from our experience with our interactions with many customers and share the best practices that is useful for other fellow Confluence administrators.
We will organise the points by 5 important considerations for Confluence Administration
- Integrity
- Security
- Performance
- Cost
- Uptime
In each section, we will explain on the implications of attachments on each aspect and recommend solutions to address them.
1) Integrity
Missing Attachments
Have you encountered the scenario where you tried downloading an attachment and got the Attachment File Not Found error message?
A system is useless if you cannot retrieve the information stored in them. Without the trust, users will not have the confidence to store their work in the system.
There are many possibilities that attachments can go missing in Confluence
- Deleted by antivirus on the backend
- Error during the uploads or blocked by the Web Application Firewalls (WAF)
- Files upload when Confluence does not have sufficient disk space
- Human errors during backup/restore during migration of servers
- Ransomware
To address the issue, we recommend to use the Missing Attachment Scanner periodically to scan your Confluence instance. It will run a full scan of your entire Confluence site during off-peak hours to see if any attachments are missing. You can also run this integrity check before migrating to Atlassian Cloud.
For those Confluence servers with anti-virus software installed, we also recommend to enable Missing File Feedback feature with Attachment Checker. It will double check if the file is accessible upon every attachment upload.
In a normal circumstance, the virus scanner will quarantine the infected file quietly. There is no feedback provided to the end users. Nobody will know the file is missing until someone tries to download it. The app helps to address this scenario by posting a comment on the Confluence page to inform the users, so that they can take timely corrective actions.
Overwritten Files
Another scenario is multiple users working on the attachment at the same time, and overwritting the newer version of attachments with an older version. Cenote Lockpoint is a Confluence app which solves the issue with a mechanism to check out attachments for exclusive editing.
Missing Metadata
In some rare scenarios, it is possible for attachments to have missing metadata (e.g. creation date and author). This is an issue when importing the data over to Confluence Cloud. Attachment Checker helps to check during the upload and also identify the list of affected files from the Missing Attachments Scanner report.
2) Security
Malicious Files
A common security weakness for web applications is CWE-434 (Unrestricted Upload of File with Dangerous Type).
A malicious file can compromise the security in 2 possible ways
- The Confluence server processes the file which results in unwanted code execution within the server itself
- Users download the file onto their computers causing a virus infection
Therefore, we recommend to implement a list of safe file extensions which is safe for Confluence.
For those Confluence sites with public users, the MIME type checks provide an additional level of security against malicious users who rename the file extension to bypass the file type checks.
Information Disclosure
Another security risk is unintentional information disclosure or wiki leaks. Sometimes an intern or external vendor may download all the attachments for purposes other than work.
While the easiest way is to secure the Confluence spaces with the correct permissions, it is also possible to manage these groups by
- restricting them from downloading files from Confluence
- keeping a log of the download activities within the space
3) Performance
Processing of large attachments
One of my favourite feature in Confluence is the ability to preview documents directly without having to download it and open with another application. However the document conversion process with very big files may cause performance issues in some cases.
When you insert a file into a page (for example a Word document, or Excel spreadsheet), Confluence will convert the contents to a format that can be viewed inline in the page, in the preview, or in some macros. This can be quite memory and CPU intensive, and has been known to cause out of memory errors when processing very complex files.
We had a customer who has encountered irresponsive Confluence on several occassions due to users uploading certain type of files. We developed the Large Attachment Tracker to facilitate the Confluence admins to do a quick check if this is a cause whenever users are reporting a slowdown.
Streaming of Media Content
If you are using Confluence as a corporate intranet or learning management system, Confluence may experience slowdown after a major corporate event when everyone is simultaneously checking out the event videos and photos from the server.
Confluence is not a video streaming server, so it may not be able to handle very high workload when a lot of users are downloading large videos at the same time.It is a best practice to split the photos and videos into several pages and turn off autoplay so that they do not hoard up a lot of resources within a single page load.
Anti-Virus Scanners
Another common reason for Confluence slowing down is due to the virus scanning. The CPU and disk I/O can increase due to inspection of files. Atlassian has put up a KB article on the best practices and workarounds when Confluence is suffering a performance issue.
A possible solution is to check each file once during the upload. This reduces the unnecessary checks during subsequent file access. It is possible by integrating with a compatible virus scanner and queuing all the uploaded attachments for a scan without overwhelming the server resources.
4) Cost
For large Confluence sites, it is a never ending uphill challenge. People are uploading attachments everyday but the disk space is finite. Without taking any action, the disk space will eventually be full.
Most people will say increasing the disk storage is a small problem since disk storage is very affordable nowadays.
Types of Hidden Costs
However, the hidden truth is the real costs is more than buying a bigger hard disk. There are a few types of costs.
Type of Cost How it affects Backup cost The amount of disk space used is even higher since it is a common practice to keep multiple generations of backups Bandwidth cost This may not be applicable for everyone. We also have a customer whose users are working on ships and their Internet bandwidth is limited and expensive. Hence they want all their images to be scaled down instead of the high resolution quality which is a norm nowadays. Operational cost This is an invisible cost in terms of energy consumption and time that system engineers spent on
– increasing the disk storage
– managing the backups
– doing upgrades and reindexing
– executing virus scans
– migrating to new hardware
– generating reports on disk usage by Confluence spacesStorage cost There is a need to upgrade to a bigger hard disk.
For those planning to migrate to Confluence Cloud, it is needed to upgrade from the Standard plan to the Premium plan once the disk usage hits a limit of 250Gb.Usage cost For larger files, it takes slightly more time to download and open.
– Every user takes 5 more seconds for each download
– A typical user downloads 5 such files a day
– A company with 500 users can save 3.4 hours a day or 104 hours a month
When the disk space is insufficient, they need to spend time to do housekeeping.There are 2 schools of thought on how to address the challenge of ever growing attachments.
Removing useless content
The first approach is to remove those content that is no longer in use. There are 2 apps on Atlassian Marketplace which allow users to identify unused attachments and bulk deletion:
Admins can also use retention rules to delete historical versions of attachments. However it is risky when some old versions contains important data.
Reducing unnecessary growth
Another approach is to prevent the hyper growth of disk usage by curtailing the uploads of very big files and unnecessary files.
The Attachment Checker provides Confluence admins a summary to identify which teams are using a lot of disk space.
With the info, Confluence admins can identify misuse as well as invalid file types to block from Confluence.
It is also possible to enforce the quota to warn or prevent users from additional uploads until they housekeep the unnecessary large files.
Likewise, space admins and users can check out the usage of their spaces when they need to do some housekeeping.
There is another guide on How to free up disk space on Confluence with more details.
5) Uptime
Lastly, when the total size of attachments grows, it takes a longer time to execute backups and upgrades.
This implies a longer downtime for scheduled maintenance activitiesConclusion
Although this article may be more relevant for bigger or enterprise scale Confluence instances, it is useful to start addressing the issues early than to spend more effort doing the cleanup in the future.
Share this post
-
CVE-2022-26134 – How to check and protect your Confluence
Last Friday, Volexity published a zero day exploit (CVE-2022-26134) on Atlassian Confluence. This post is to share some tips on how to check your Confluence instance is safe, and also some practical advice to protect your Confluence on-prem.
About the vulnerability
This bug affects all versions of Confluence since 1.3.0. It is a critical vulnerability because it allows unauthenticated users to execute code within the Confluence server remotely. According to Imperva Threat Research, there are widespread scanning and attempts of exploitation on the Internet.
How to fix the vulnerability
Atlassian alerted the customers promptly and responded with high priority. We are thankful that Atlassian released the fix in less than 24 hours.
For details of the fix, please refer to the official Confluence Security Advisory 2022-06-02.
How to check your Confluence for malicious access
Here are some basic checks that you can execute to check for any traces of malicious attempts. If there is any occurrence, then you may want to engage the security experts for more in-depth foresenic investigation.
URL requests containing ${
Since one of the attack mechanisms is to use ${ in the request URL, it would be helpful to scan the web server access logs for any occurrences. Please update the path of the Apache httpd/ Nginx access logs accordingly.
grep '${' /etc/httpd/logs/*access*.log grep '%24%7B' /etc/httpd/logs/*access*.log
URL requests from known IP addresses
Based on the Volexity report, there are some IP addresses which are used by the attackers. Similarly, you can grep the access logs to check for any occurrences. Note: It is possible that there may be other attackers using other IP addresses.
grep 154.146.34.145 /etc/httpd/logs/*access* grep 154.16.105.147 /etc/httpd/logs/*access* grep 156.146.34.46 /etc/httpd/logs/*access* grep 156.146.34.52 /etc/httpd/logs/*access* grep 156.146.34.9 /etc/httpd/logs/*access* grep 156.146.56.136 /etc/httpd/logs/*access* grep 198.147.22.148 /etc/httpd/logs/*access* grep 198.147.22.148 /etc/httpd/logs/*access* grep 221.178.126.244 /etc/httpd/logs/*access* grep 45.43.19.91 /etc/httpd/logs/*access* grep 59.163.248.170 /etc/httpd/logs/*access* grep 64.64.228.239 /etc/httpd/logs/*access* grep 66.115.182.102 /etc/httpd/logs/*access* grep 66.115.182.111 /etc/httpd/logs/*access* grep 67.149.61.16 /etc/httpd/logs/*access* grep 98.32.230.38 /etc/httpd/logs/*access*
How to protect your Confluence instance
Actually, the best form of defense against unauthenticated attacks is to place the server behind the firewall. This will effectively block all attackers from mounting a direct attack remotely. That is a key reason why some security sensitive enterprises are choosing Confluence Data Center. We know that it is not possible for a software to be 100% free of bugs. So there might be another vulnerability waiting to be discovered in the future.
By using Long Term Support release of the product, it reduces the effort to upgrade since the critical security fixes will be available as long it is architecturally possible. This contributes greatly to a quick reaction to any future zero day exploits.
For those organizations who are working remotely, it is possible to access via VPN or use Web application firewalls for added protection. Both CloudFlare and Imperva have announced that their customers are protected from this vulnerability since they will ensure all requests are authenticated before relaying it to Confluence.
Last but not least, do make sure the license technical contacts are up-to-date. As an Atlassian Solution Partner, we have witnessed a number of occurrences when critical alerts from Atlassian are missed due to staff turnover.
Share this post