Attachments are a special class citizen in Atlassian Confluence but a lot of Confluence admins do not pay attention on them until bigger problem starts to surface.
This comprehensive guide comes from our experience with our interactions with many customers and share the best practices that is useful for other fellow Confluence administrators.
We will organise the points by 5 important considerations for Confluence Administration
Integrity
Security
Performance
Cost
Uptime
In each section, we will explain on the implications of attachments on each aspect and recommend solutions to address them.
1) Integrity
Missing Attachments
Have you encountered the scenario where you tried downloading an attachment and got the Attachment File Not Found error message?
A system is useless if you cannot retrieve the information stored in them. Without the trust, users will not have the confidence to store their work in the system.
There are many possibilities that attachments can go missing in Confluence
Deleted by antivirus on the backend
Error during the uploads or blocked by the Web Application Firewalls (WAF)
Files upload when Confluence does not have sufficient disk space
Human errors during backup/restore during migration of servers
Ransomware
To address the issue, we recommend to use the Missing Attachment Scanner periodically to scan your Confluence instance. It will run a full scan of your entire Confluence site during off-peak hours to see if any attachments are missing. You can also run this integrity check before migrating to Atlassian Cloud.
For those Confluence servers with anti-virus software installed, we also recommend to enable Missing File Feedback feature with Attachment Checker. It will double check if the file is accessible upon every attachment upload.
In a normal circumstance, the virus scanner will quarantine the infected file quietly. There is no feedback provided to the end users. Nobody will know the file is missing until someone tries to download it. The app helps to address this scenario by posting a comment on the Confluence page to inform the users, so that they can take timely corrective actions.
Overwritten Files
Another scenario is multiple users working on the attachment at the same time, and overwritting the newer version of attachments with an older version. Cenote Lockpoint is a Confluence app which solves the issue with a mechanism to check out attachments for exclusive editing.
Missing Metadata
In some rare scenarios, it is possible for attachments to have missing metadata (e.g. creation date and author). This is an issue when importing the data over to Confluence Cloud. Attachment Checker helps to check during the upload and also identify the list of affected files from the Missing Attachments Scanner report.
2) Security
Malicious Files
A common security weakness for web applications is CWE-434 (Unrestricted Upload of File with Dangerous Type).
A malicious file can compromise the security in 2 possible ways
The Confluence server processes the file which results in unwanted code execution within the server itself
Users download the file onto their computers causing a virus infection
For those Confluence sites with public users, the MIME type checks provide an additional level of security against malicious users who rename the file extension to bypass the file type checks.
Information Disclosure
Another security risk is unintentional information disclosure or wiki leaks. Sometimes an intern or external vendor may download all the attachments for purposes other than work.
While the easiest way is to secure the Confluence spaces with the correct permissions, it is also possible to manage these groups by
restricting them from downloading files from Confluence
keeping a log of the download activities within the space
3) Performance
Processing of large attachments
One of my favourite feature in Confluence is the ability to preview documents directly without having to download it and open with another application. However the document conversion process with very big files may cause performance issues in some cases.
When you insert a file into a page (for example a Word document, or Excel spreadsheet), Confluence will convert the contents to a format that can be viewed inline in the page, in the preview, or in some macros. This can be quite memory and CPU intensive, and has been known to cause out of memory errors when processing very complex files.
We had a customer who has encountered irresponsive Confluence on several occassions due to users uploading certain type of files. We developed the Large Attachment Tracker to facilitate the Confluence admins to do a quick check if this is a cause whenever users are reporting a slowdown.
Streaming of Media Content
If you are using Confluence as a corporate intranet or learning management system, Confluence may experience slowdown after a major corporate event when everyone is simultaneously checking out the event videos and photos from the server. Confluence is not a video streaming server, so it may not be able to handle very high workload when a lot of users are downloading large videos at the same time.
It is a best practice to split the photos and videos into several pages and turn off autoplay so that they do not hoard up a lot of resources within a single page load.
Anti-Virus Scanners
Another common reason for Confluence slowing down is due to the virus scanning. The CPU and disk I/O can increase due to inspection of files. Atlassian has put up a KB article on the best practices and workarounds when Confluence is suffering a performance issue.
A possible solution is to check each file once during the upload. This reduces the unnecessary checks during subsequent file access. It is possible by integrating with a compatible virus scanner and queuing all the uploaded attachments for a scan without overwhelming the server resources.
4) Cost
For large Confluence sites, it is a never ending uphill challenge. People are uploading attachments everyday but the disk space is finite. Without taking any action, the disk space will eventually be full.
Most people will say increasing the disk storage is a small problem since disk storage is very affordable nowadays.
Types of Hidden Costs
However, the hidden truth is the real costs is more than buying a bigger hard disk. There are a few types of costs.
Type of Cost
How it affects
Backup cost
The amount of disk space used is even higher since it is a common practice to keep multiple generations of backups
Bandwidth cost
This may not be applicable for everyone. We also have a customer whose users are working on ships and their Internet bandwidth is limited and expensive. Hence they want all their images to be scaled down instead of the high resolution quality which is a norm nowadays.
Operational cost
This is an invisible cost in terms of energy consumption and time that system engineers spent on – increasing the disk storage – managing the backups – doing upgrades and reindexing – executing virus scans – migrating to new hardware – generating reports on disk usage by Confluence spaces
Storage cost
There is a need to upgrade to a bigger hard disk. For those planning to migrate to Confluence Cloud, it is needed to upgrade from the Standard plan to the Premium plan once the disk usage hits a limit of 250Gb.
Usage cost
For larger files, it takes slightly more time to download and open. – Every user takes 5 more seconds for each download – A typical user downloads 5 such files a day – A company with 500 users can save 3.4 hours a day or 104 hours a month When the disk space is insufficient, they need to spend time to do housekeeping.
There are 2 schools of thought on how to address the challenge of ever growing attachments.
Removing useless content
The first approach is to remove those content that is no longer in use. There are 2 apps on Atlassian Marketplace which allow users to identify unused attachments and bulk deletion:
Lastly, when the total size of attachments grows, it takes a longer time to execute backups and upgrades. This implies a longer downtime for scheduled maintenance activities
Conclusion
Although this article may be more relevant for bigger or enterprise scale Confluence instances, it is useful to start addressing the issues early than to spend more effort doing the cleanup in the future.
What is your biggest Jira concern? Accidental deletion of data from Jira, or data loss during migration?
The bigger a supporter of Jira you are, the more you know that Atlassian does not ensure you with granular, point-in-time restore in case of unintentional deletion and daily operations.
Register to join a special launch webinar of our Partner GitProtect.io, introducing Jira Granular Restore technology on Nov 16, at 10 AM PT / 6 PM CET, and find out how to:
Migrate Jira data and the entire configuration from one project to another
Instantly restore any deleted object in Jia – projects, issues, workflows, attachments
Move between various Jira accounts
Copy project configurations from Sandbox to Production
Separate out projects or consolidate different Jira sites into one
LIVE DEMO – GitProtect Jira Backup and Granular Restore
P.S. A dedicated webinar recording will be sent to registered people who will not be able to participate in the recording.
external system sending thousands of REST API calls (to retrieve or update update)
incompatibility across plugins
Individually each app works ok, but they cause issues with each other
CPU intensive jobs
e.g. project level exports can be intensive if there are a lot of custom fields
a lot of custom fields
By updating the custom field context, it can help to improve the performance
The best way to improve the speed is to identify the root cause and treat the problem accordingly.
If it is a plugin related issue, you can try to disable the plugin to isolate the problem
If it is a data triggered issue, you can try to remove the data
You can use Site Statistics for Jira Cloud to track the historical growth of issues in your Cloud instance. It can be helpful to see if the slowness is caused by huge increase of issues.
What does Atlassian Intelligence says?
I posed the question “Possible reasons causing Jira Cloud to be slow” and got the answer below
High network latency: Slow internet connection or high network congestion can result in slower response times for Jira Cloud.
Large number of issues: If your Jira project contains a large number of issues, it may impact the performance of Jira Cloud.
Heavy usage: Concurrently performing resource-intensive operations such as searching, reporting, or updating a large number of issues can slow down Jira Cloud.
Misconfigured plugins: In some cases, misconfigured or poorly optimized plugins can cause performance issues in Jira Cloud.
Insufficient hardware resources: Inadequate server resources such as CPU, memory, or disk space can affect the performance of Jira Cloud. ❌
Outdated software version: Running an outdated version of the Jira application may lead to performance degradation and compatibility issues with other components.❌
To troubleshoot and improve the performance of your Jira Cloud instance, you can follow these steps:
Check your network connectivity and ensure that you have a stable internet connection.
Optimize your issue database by archiving or deleting unnecessary data to reduce database size and improve overall system performance.
Review installed plugins and disable any that are not essential or causing slowdowns. Ensure that all installed plugins are up-to-date with compatible versions for your current JIRA setup.
Monitor system resources (CPU, memory) on the server running JIRA to identify any bottlenecks and consider scaling up hardware if required.
Regularly update your JIRA application to benefit from bug fixes and performance improvements provided by Atlassian through new releases.
❌ The hardware for Jira Cloud is managed by Atlassian. It is always upgraded to the latest version automatically.
Banks are well known for their stringent policies on security and compliance. This is because they need to ensure their customers’ data are well protected.
We scouted the Internet to research which banks are using Atlassian Cloud and compiled the list below.
Bank
Country / Region
Product(s)
Solution Partner
User Count
Latin America
Jira
eCore
4,300
Mexico
Confluence, Jira, JSM
bit2bit Americas
500
Commonwealth Bank
Australia
Confluence, Jira
25,000
EQ Bank
Canada
Confluence, Jira
Blended Perspectives
400
Hanseatic Bank
Germany
Confluence, Jira
Jodocus GmbH
Libra Bank
Romania
Jira
Life in Codes
p.s: The list is ordered by the name of the bank, followed by Country/Region
Hope the info will be useful for financial institutions who are exploring to migrate to Jira Cloud.
Are you curious which apps other users are buying to extend the capabilities of their Jira, Confluence or Bitbucket?
This year, we are pleased to share again our updated Top 10 Popular apps with fellow Atlassian users. It is a good opportunity to review which useful capabilities to add to your Atlassian suite.
From our perspective, Marketplace apps play a significant role for successful adoption by
enabling automation to improve productivity, speed or security
providing additional capabilities like Business Analytics, Test Automation, etc
organising information to provide insight and facilitate collaboration
How is the ranking done?
The ranking is based on the number of licenses bought through us in 2022. We felt this will be a better measure of the popularity of the app.
In event of a tie, we go by the licensed users count, followed by the total sale value for the app.
Akeles Top 10 List
Congratulations to the winners. It is an achievement given there are over 4,300 apps listed in Atlassian Marketplace.
Do you know that July is the Jira Admin Appreciation Month, and 15th of July is the official Jira Admin Appreciation Day?
As an Atlassian Solution Partner as well as a Marketplace Partner, we work closely with many Jira Admins and witnessed their passion, ingenuity and dedication on countless occasions.
We would like to take the opportunity to share some wonderful Jira Admins we encountered
Kamar who worked with us to troubleshoot a mystery case on the sudden slowdown in Jira’s performance
Jun Xiang who set up a new service desk project all by himself, saving the money to buy an additional system
Hany who suggested improvements for a Marketplace app so that his team can work more effectively
Graeme who organised lunch and learn sessions for colleagues to share his Jira knowledge
Coral who stayed up until 5am so that Jira can be operational when her colleagues return to work on Monday
and many others who took time after work to attend Atlassian Community Events to beef up their knowledge
What people think a Jira Admin do?
Going by the literal meaning, the Jira administrator is the person who administers the Jira web application.
What a Jira Admin really do?
However in the real life, the Jira Admins are responsible for everything that is related to Jira. This is a norm because many organisations do not have a team to manage Jira. Usually the Jira admin will have to wear multiple hats. More importantly, these roles also require knowledge of Jira.
Here are some additional roles the Jira Admins are taking up:
Jira System Engineer
This role focuses on tasks related with systems. It requires competency in both inner workings of Jira as well as the backend systems. Some examples of the tasks are:
Handle Level 2 support by analysing Jira application or access logs
Work with Atlassian Support or App Vendors for complex cases
Using SQL on the database to generate reports or patch data
Perform Application/Server Performance Tuning
Perform upgrades and Disaster Recovery (DR) planning
Work with Security to conduct Vulnerability Assessment & Penetration Testing (VAPT)
Jira Solution Engineer
This role focuses on the business aspect. By providing solutions using Jira to deliver new capabilities, it increases the ROI. Some examples of the tasks are:
Create Jira project templates for new use cases
Build Jira workflows that help to improve the flow
Design Jira dashboards or BI reports to give visibility to the stakeholders
Select Marketplace apps to fulfill business requirements or improve productivity
Write scripts to automate some tasks
Or even coding Jira plugins for customised features
Jira Coach
This role focuses on the people aspect by helping fellow Jira users to use Jira more effectively. Some examples of the tasks are:
Conduct training
Answer questions related on the usage
Write KB articles on Confluence
Promote the use of Jira within the organisation
Analyse statistics to identify trends and area for improvement
How to help your Jira Admins?
In some scenarios, the Jira admin might even be a part-time responsibility in additional to their official job description.
The workload will pile up until the company will engage a Solution Partner or an Atlassian Technical Account Manager for additional support.
We have listed 9 ways to reduce the workload for your beloved Jira Admins
Give up on your Jira admin rights (if you are not trained in Jira)
That can reduce unnecessary fire-fighting due to mistakes
Otherwise get proper training to be a Jira admins
Look for the Jira project admins instead of the Jira admins for project permission requests
It can be death by a thousand paper cuts with 1 request from every user
Standardize your project workflows
It can be messy when every project have a different workflow and different set of custom fields
Raise your requests in Jira
That will facilitate tracking and fulfilment by the Jira Admins
Use apps
They can automate some of the manual tasks taking up the Jira Admin’s time
Use a LTS version to reduce the upgrade cadence
Every upgrade consumes time and effort
It is easier to patch an LTS version
It helps to minimise the turnaround time in event of a security advisory
Upgrade at least once a year
The risk, complexity and technical debt increases over time
Host Jira behind the firewall
Use VPN or Zero Trust Network to access if your team are working remotely
That will reduce a lot of work on security
Use Jira Cloud if it is suitable for your organisation
Atlassian will take over some of the workload
Hopefully with more time, the Jira admins can make Jira better for everyone.
Last Friday, Volexity published a zero day exploit (CVE-2022-26134) on Atlassian Confluence. This post is to share some tips on how to check your Confluence instance is safe, and also some practical advice to protect your Confluence on-prem.
About the vulnerability
This bug affects all versions of Confluence since 1.3.0. It is a critical vulnerability because it allows unauthenticated users to execute code within the Confluence server remotely. According to Imperva Threat Research, there are widespread scanning and attempts of exploitation on the Internet.
How to fix the vulnerability
Atlassian alerted the customers promptly and responded with high priority. We are thankful that Atlassian released the fix in less than 24 hours.
Here are some basic checks that you can execute to check for any traces of malicious attempts. If there is any occurrence, then you may want to engage the security experts for more in-depth foresenic investigation.
URL requests containing ${
Since one of the attack mechanisms is to use ${ in the request URL, it would be helpful to scan the web server access logs for any occurrences. Please update the path of the Apache httpd/ Nginx access logs accordingly.
Based on the Volexity report, there are some IP addresses which are used by the attackers. Similarly, you can grep the access logs to check for any occurrences. Note: It is possible that there may be other attackers using other IP addresses.
Actually, the best form of defense against unauthenticated attacks is to place the server behind the firewall. This will effectively block all attackers from mounting a direct attack remotely. That is a key reason why some security sensitive enterprises are choosing Confluence Data Center. We know that it is not possible for a software to be 100% free of bugs. So there might be another vulnerability waiting to be discovered in the future.
By using Long Term Support release of the product, it reduces the effort to upgrade since the critical security fixes will be available as long it is architecturally possible. This contributes greatly to a quick reaction to any future zero day exploits.
For those organizations who are working remotely, it is possible to access via VPN or use Web application firewalls for added protection. Both CloudFlare and Imperva have announced that their customers are protected from this vulnerability since they will ensure all requests are authenticated before relaying it to Confluence.
Last but not least, do make sure the license technical contacts are up-to-date. As an Atlassian Solution Partner, we have witnessed a number of occurrences when critical alerts from Atlassian are missed due to staff turnover.
From our perspective, Marketplace apps play a significant role for successful adoption of Atlassian platforms by
enabling automation to improve productivity, speed or security
adding features to provide additional capabilities like Business Analytics, Test Automation, etc
organising information to provide insight and facilitate collaboration
This year, Atlassian Marketplace reached $2 billion in lifetime sales. This is a huge testimony of the usefulness and popularity of Marketplace apps.
How is the ranking done?
The ranking is based on the number of licenses (Server/DC/Cloud) customers bought in 2021. We felt this will be a better measure of the popularity of the app.
In event of a tie, we go by the licensed users count, followed by the total sale value for the app.
Akeles Top 10 List
We are pleased to share our list for 2021 voted by the Atlassian users in Singapore. Although our list may not correspond to the global popularity in Atlassian Marketplace, it is an affirmation in the usefulness of the apps.
CVE-2021-44228 or log4shell is a serious vulnerability discovered recently. It allows an attacker to execute malicious code in any applications which uses a vulnerable version of log4j (Version 2.0 onwards). The impact is very severe because:
It is extremely simple to execute such an attack
Log4J is the most popular logging framework used by many Java applications
There are already many attempts on the Internet to scan for this loophole
The official guideline is to patch the applications to upgrade to Log4J version 2.16 onwards.
Are Akeles Jira/Confluence apps safe?
Thankfully with applications like Bitbucket and Sonatype Nexus Lifecycle, we were able to identify the 3rd party components used in our applications.
We have verified that we do not bundle the log4j library in our Jira/Confluence apps. We are using the log4j library that is provided by Jira/Confluence. Hence we are safe.
Another piece of reassuring news is Atlassian is also scanning the apps listed on the Atlassian Marketplace.
Each vulnerable DC or server app will be given the same expedited deadline as cloud apps. DC and server apps that fail to address the vulnerability within this expedited timeframe will be removed from the marketplace, and then Atlassian will inform customers who have vulnerable paid apps installed.
Atlassian has put up a detailed official advisory that stated that Jira and Confluence are using an Atlassian-maintained fork of Log4J 1.2.17 which is not vulnerable to CVE-2021-44228. However they confirmed a similar but low risk vulnerability (CVE-2021-4104) which is exposed only if the log4j configuration has been modified from their default settings.
The risk is low because these settings are not enabled by default. Nevertheless, it is better to counter check again.
For our customers who are using Sonatype products, Nexus Lifecycle, Nexus Firewall or Nexus Repository are using logback instead of log4j. Hence they are not affected. The official statement is available at Sonatype website.
Food for Thoughts
The connectivity of the Internet makes it even more challenging to prevent such zero-day vulnerability. Here are some questions we need to take in considerations for the IT strategy (tooling, SaaS services, architecture, processes, automation, etc)
How can we be notified of any vulnerabilities as soon as possible?
How can we minimise the risks and impact of an attack?
How can we identify the affected applications quickly?
How can we ensure the 3rd party libraries used are safe?
How can we patch the affected applications in a timely manner?
Useful Resources
You may want to consult the following pages for more information
One of Jira’s strengths is that it allows 3rd party integration via REST API calls. By providing the username, password and Base URL, it is possible to 3rd party apps to query or update Jira automatically.
In this article, we will share why using API Tokens is a better and safer option than using Password Authentication.
The Power of Passwords
Besides entering passwords on the Jira login screen, it is also possible to provide the passwords on 3rd party applications or scripts to execute REST API calls.
Some of the use cases are like
Create issues from Slack
Send alerts to Microsoft Teams
Update Jira issues with Commits information from GitHub
Integrate with your in-house systems
If the password fails in the wrong hands, it is possible that
Wikileaks of your confidential data
Your Jira system can slow down drastically due to excessive API calls which affects the usage of other users
Benefits of using API Tokens over Passwords
By using API Token, it improves the security of your Jira instance
Safer – The API Token has a certain level of password complexity which defends against dictionary attacks
Isolation – It distributes the risk by having a different API token for each 3rd party integration. It is possible to revoke/reset the token for that application without any impact to other applications.
Differentiation – With a different mechanism, it is possible to apply more stringent checks on the usage of API Tokens (e.g. restriction by IP address range)
Control – It restricts ordinary users from using their credentials to do REST API calls
Availability – For sites running on Single Sign On. Users will not know their passwords other than their Windows passwords
Validity – It is possible to set the expiry date of the token
It allows basic authentication with API Tokens. Currently, it is not possible to use the Jira Data Center’s Personal Access Token together with the username on 3rd party websites.
Warning: If you disable Basic Authentication with passwords in the System Wide settings, you also can’t authenticate on non REST endpoints with API Tokens directly. You can still do that by reusing a session you got from authenticating with an API Token.
Able to limit usage to particular IP ranges
It is possible to limit the usage of the API token to the IP address of the internal system. You can ensure the REST API calls are coming from your trusted network.
Block requests with malicious characters in path
This is a bonus feature which helps to defend against some attack vectors.
Limit usage of API Tokens
It is a security best practice to grant rights only to users who needs it and has proper training. There are incidents arising from users who entered their Jira passwords on 3rd party sites or executed a buggy script.
Tip: We recommend to create a group “jira-api-users” to manage those service account users who can use API Tokens.
Set a validity of the API Token
If the token is for testing or for temporary usage, the Jira admin can just set a shorter validity that will expire automatically. Otherwise it relies on the Jira Admins to remember to revoke the access manually.
Service Accounts typically do not have a password validity. If the service account is from an Active Directory, there could be disruption if there is a 90 day reset password policy.
By using the Active Directory passwords, it is possible that account is locked out of all applications after multiple wrong password attempts.
Control over audit logging
It offers admins a fine level of control over the information to be logged.
