• Best Practices in Confluence Administration – Attachments

    Best Practices in Confluence Administration for Attachments

    Introduction

    Attachments are a special class citizen in Atlassian Confluence but a lot of Confluence admins do not pay attention on them until bigger problem starts to surface.

    This comprehensive guide comes from our experience with our interactions with many customers and share the best practices that is useful for other fellow Confluence administrators.

    We will organise the points by 5 important considerations for Confluence Administration

    1. Integrity 
    2. Security 
    3. Performance
    4. Cost
    5. Uptime

    In each section, we will explain on the implications of attachments on each aspect and recommend solutions to address them.

    1) Integrity

    Missing Attachments

    Have you encountered the scenario where you tried downloading an attachment and got the Attachment File Not Found error message?

    Attachment File Not Found error

    A system is useless if you cannot retrieve the information stored in them. Without the trust, users will not have the confidence to store their work in the system.

    There are many possibilities that attachments can go missing in Confluence

    • Deleted by antivirus on the backend
    • Error during the uploads or blocked by the Web Application Firewalls (WAF)
    • Files upload when Confluence does not have sufficient disk space
    • Human errors during backup/restore during migration of servers
    • Ransomware

    To address the issue, we recommend to use the Missing Attachment Scanner periodically to scan your Confluence instance. It will run a full scan of your entire Confluence site during off-peak hours to see if any attachments are missing. You can also run this integrity check before migrating to Atlassian Cloud.

    Missing Attachment Scanner checking for missing attachments

    For those Confluence servers with anti-virus software installed, we also recommend to enable Missing File Feedback feature with Attachment Checker. It will double check if the file is accessible upon every attachment upload.

    In a normal circumstance, the virus scanner will quarantine the infected file quietly. There is no feedback provided to the end users. Nobody will know the file is missing until someone tries to download it. The app helps to address this scenario by posting a comment on the Confluence page to inform the users, so that they can take timely corrective actions.

    Alert to the Confluence user through a comment when the virus scanner detects an infected file

    Overwritten Files

    Another scenario is multiple users working on the attachment at the same time, and overwritting the newer version of attachments with an older version. Cenote Lockpoint is a Confluence app which solves the issue with a mechanism to check out attachments for exclusive editing.

    Missing Metadata

    In some rare scenarios, it is possible for attachments to have missing metadata (e.g. creation date and author). This is an issue when importing the data over to Confluence Cloud. Attachment Checker helps to check during the upload and also identify the list of affected files from the Missing Attachments Scanner report.

    Alerts when attachment does not have a creation date

    2) Security

    Malicious Files

    A common security weakness for web applications is CWE-434 (Unrestricted Upload of File with Dangerous Type).

    A malicious file can compromise the security in 2 possible ways

    1. The Confluence server processes the file which results in unwanted code execution within the server itself
    2. Users download the file onto their computers causing a virus infection

    Therefore, we recommend to implement a list of safe file extensions which is safe for Confluence.

    Configure the file types allowed or blocked

    For those Confluence sites with public users, the MIME type checks provide an additional level of security against malicious users who rename the file extension to bypass the file type checks.

    2 layer file check so that renaming the extension cannot trick the app

    Information Disclosure

    Another security risk is unintentional information disclosure or wiki leaks. Sometimes an intern or external vendor may download all the attachments for purposes other than work.

    While the easiest way is to secure the Confluence spaces with the correct permissions, it is also possible to manage these groups by

    • restricting them from downloading files from Confluence
    • keeping a log of the download activities within the space
    Keep a log when users download non image files

    3) Performance

    Processing of large attachments

    One of my favourite feature in Confluence is the ability to preview documents directly without having to download it and open with another application. However the document conversion process with very big files may cause performance issues in some cases.

    When you insert a file into a page (for example a Word document, or Excel spreadsheet), Confluence will convert the contents to a format that can be viewed inline in the page, in the preview, or in some macros. This can be quite memory and CPU intensive, and has been known to cause out of memory errors when processing very complex files.

    We had a customer who has encountered irresponsive Confluence on several occassions due to users uploading certain type of files. We developed the Large Attachment Tracker to facilitate the Confluence admins to do a quick check if this is a cause whenever users are reporting a slowdown.

    Display the list of large files uploaded recently

    Streaming of Media Content

    If you are using Confluence as a corporate intranet or learning management system, Confluence may experience slowdown after a major corporate event when everyone is simultaneously checking out the event videos and photos from the server.
    Confluence is not a video streaming server, so it may not be able to handle very high workload when a lot of users are downloading large videos at the same time.

    It is a best practice to split the photos and videos into several pages and turn off autoplay so that they do not hoard up a lot of resources within a single page load.

    Anti-Virus Scanners

    Another common reason for Confluence slowing down is due to the virus scanning. The CPU and disk I/O can increase due to inspection of files. Atlassian has put up a KB article on the best practices and workarounds when Confluence is suffering a performance issue.

    A possible solution is to check each file once during the upload. This reduces the unnecessary checks during subsequent file access. It is possible by integrating with a compatible virus scanner and queuing all the uploaded attachments for a scan without overwhelming the server resources.

    integration with 3rd party virus scanners to check when attachment is uploaded to Confluence

    4) Cost

    For large Confluence sites, it is a never ending uphill challenge. People are uploading attachments everyday but the disk space is finite. Without taking any action, the disk space will eventually be full.

    Most people will say increasing the disk storage is a small problem since disk storage is very affordable nowadays.

    Types of Hidden Costs

    However, the hidden truth is the real costs is more than buying a bigger hard disk. There are a few types of costs.

    Type of CostHow it affects
    Backup costThe amount of disk space used is even higher since it is a common practice to keep multiple generations of backups
    Bandwidth costThis may not be applicable for everyone. We also have a customer whose users are working on ships and their Internet bandwidth is limited and expensive. Hence they want all their images to be scaled down instead of the high resolution quality which is a norm nowadays.
    Operational costThis is an invisible cost in terms of energy consumption and time that system engineers spent on
    – increasing the disk storage
    – managing the backups
    – doing upgrades and reindexing
    – executing virus scans
    – migrating to new hardware
    – generating reports on disk usage by Confluence spaces
    Storage costThere is a need to upgrade to a bigger hard disk.
    For those planning to migrate to Confluence Cloud, it is needed to upgrade from the Standard plan to the Premium plan once the disk usage hits a limit of 250Gb.
    Usage costFor larger files, it takes slightly more time to download and open.
    – Every user takes 5 more seconds for each download
    – A typical user downloads 5 such files a day
    – A company with 500 users can save 3.4 hours a day or 104 hours a month
    When the disk space is insufficient, they need to spend time to do housekeeping.

    There are 2 schools of thought on how to address the challenge of ever growing attachments.

    Removing useless content

    The first approach is to remove those content that is no longer in use. There are 2 apps on Atlassian Marketplace which allow users to identify unused attachments and bulk deletion:

    Admins can also use retention rules to delete historical versions of attachments. However it is risky when some old versions contains important data.

    Reducing unnecessary growth

    Another approach is to prevent the hyper growth of disk usage by curtailing the uploads of very big files and unnecessary files.

    The Attachment Checker provides Confluence admins a summary to identify which teams are using a lot of disk space.

    Confluence admins can view and set disk space quota for Confluence spaces

    With the info, Confluence admins can identify misuse as well as invalid file types to block from Confluence.

    It is also possible to enforce the quota to warn or prevent users from additional uploads until they housekeep the unnecessary large files.

    Alert banner to inform users that disk usage is reaching the threshold

    Likewise, space admins and users can check out the usage of their spaces when they need to do some housekeeping.

    View the disk space usage for the current Confluence space

    There is another guide on How to free up disk space on Confluence with more details.

    5) Uptime

    Lastly, when the total size of attachments grows, it takes a longer time to execute backups and upgrades.
    This implies a longer downtime for scheduled maintenance activities

    Conclusion

    Although this article may be more relevant for bigger or enterprise scale Confluence instances, it is useful to start addressing the issues early than to spend more effort doing the cleanup in the future.

    Share this post

  • CVE-2022-26134 – How to check and protect your Confluence

    6 June 2022
    Comments are off for this post

    Last Friday, Volexity published a zero day exploit (CVE-2022-26134) on Atlassian Confluence. This post is to share some tips on how to check your Confluence instance is safe, and also some practical advice to protect your Confluence on-prem. 

    About the vulnerability

    This bug affects all versions of Confluence since 1.3.0. It is a critical vulnerability because it allows unauthenticated users to execute code within the Confluence server remotely. According to Imperva Threat Research, there are widespread scanning and attempts of exploitation on the Internet.

    How to fix the vulnerability

    Atlassian alerted the customers promptly and responded with high priority. We are thankful that Atlassian released the fix in less than 24 hours.

    For details of the fix, please refer to the official Confluence Security Advisory 2022-06-02.

    How to check your Confluence for malicious access

    Here are some basic checks that you can execute to check for any traces of malicious attempts. If there is any occurrence, then you may want to engage the security experts for more in-depth foresenic investigation.

    URL requests containing ${

    Since one of the attack mechanisms is to use ${ in the request URL, it would be helpful to scan the web server access logs for any occurrences. Please update the path of the Apache httpd/ Nginx access logs accordingly.

    grep '${' /etc/httpd/logs/*access*.log
    grep '%24%7B' /etc/httpd/logs/*access*.log
    

    URL requests from known IP addresses

    Based on the Volexity report, there are some IP addresses which are used by the attackers. Similarly, you can grep the access logs to check for any occurrences. Note: It is possible that there may be other attackers using other IP addresses.

    grep 154.146.34.145 /etc/httpd/logs/*access*
    grep 154.16.105.147 /etc/httpd/logs/*access*
    grep 156.146.34.46 /etc/httpd/logs/*access*
    grep 156.146.34.52 /etc/httpd/logs/*access*
    grep 156.146.34.9 /etc/httpd/logs/*access*
    grep 156.146.56.136 /etc/httpd/logs/*access*
    grep 198.147.22.148 /etc/httpd/logs/*access*
    grep 198.147.22.148 /etc/httpd/logs/*access*
    grep 221.178.126.244 /etc/httpd/logs/*access*
    grep 45.43.19.91 /etc/httpd/logs/*access*
    grep 59.163.248.170 /etc/httpd/logs/*access*
    grep 64.64.228.239 /etc/httpd/logs/*access*
    grep 66.115.182.102 /etc/httpd/logs/*access*
    grep 66.115.182.111 /etc/httpd/logs/*access*
    grep 67.149.61.16 /etc/httpd/logs/*access*
    grep 98.32.230.38 /etc/httpd/logs/*access*

    How to protect your Confluence instance

    Actually, the best form of defense against unauthenticated attacks is to place the server behind the firewall. This will effectively block all attackers from mounting a direct attack remotely. That is a key reason why some security sensitive enterprises are choosing Confluence Data Center. We know that it is not possible for a software to be 100% free of bugs. So there might be another vulnerability waiting to be discovered in the future.

    By using Long Term Support release of the product, it reduces the effort to upgrade since the critical security fixes will be available as long it is architecturally possible. This contributes greatly to a quick reaction to any future zero day exploits.

    For those organizations who are working remotely, it is possible to access via VPN or use Web application firewalls for added protection. Both CloudFlare and Imperva have announced that their customers are protected from this vulnerability since they will ensure all requests are authenticated before relaying it to Confluence.

    Last but not least, do make sure the license technical contacts are up-to-date. As an Atlassian Solution Partner, we have witnessed a number of occurrences when critical alerts from Atlassian are missed due to staff turnover.

    Share this post