How I reduced my catalog folder size by over 95%
Answered
Hi all,
after investigating the Luminar catalog folder I noticed that the biggest file space consumer was the resource folder (Luminar Catalog\CachedDocuments\resources). In my case 13,2 GB. After analyzing the files inside the folder I discovered that these files are a) LUT (.lut files) files which are used during editing and b) mask information if you use Accent AI with a brush for example (.miplcompressedmaskdata files). It looks like that Luminar generates one file for each editing step (LUT or mask). I think/guess this is done due to performance and to provide editing history steps for each picture. Fine for me but it consumes lots of disk space :(
Especially if you own a Surface/Laptop or just have a small SSD this is not really optimal. Sure, previews can be deleted (and will be created new after viewing a photo) as described here: https://community.skylum.com/hc/en-us/community/posts/360044100991-Luminar-Catalog-keeps-growing-regardless-of-photo-count-
But this will not cleanup the main consumer which is the resource folder. So I thought of solving this with a different approach. Deleting files was not an option because I would loose my editing history. I analyzed the files a second time and discovered that many files (especially LUT files) are the same, just the filename was different. So I wrote a small analyzer to create a hash for each file content and refer it to the file is was created from. I landed up with this:
This means that I had a maximum of 1557 files with the same content (I even filtered out hashes with a file count less then 30)! As mentioned above, deleting files was not option. So I thought it would be nice if I can refer all duplicates to only one file. Some kind of deduplication if you want ;) On a Windows OS this can be achieved by using soft or sym links https://en.wikipedia.org/wiki/Symbolic_link#NTFS_symbolic_link
I upgraded my Luminar 3 Helper https://community.skylum.com/hc/en-us/community/posts/360048117072-Workspace-Camera-Profiles-LUT-and-Collection-Editor with a resource cleaner to reduce the folder size. Reducing the folder is achieved with the following steps:
1. Folder/File analysis
2. Hash creation of each file
3. Sorting/Refering files to hashes
4. Creation of reference files. Those files will be the "source" of created softlinks
5. Deletion of duplicate files and creation of a softlink instead
After analyzing the folder Luminar 3 Helper will present the analysis result:
After running the cleanup my folder look like this:
Luminar will work normal (tested on different devices with Windows 10 1903 latest updates). Luminar uses .Net framework and therefore follows "Microsoft rules" of accessing files which makes the trick work perfectly ;)
Here is a short demo video https://youtu.be/LrAdnAa8LfQ
Download: https://www.mediafire.com/file/fxe608kznq9vhpq/Luminar_Helper.zip/file
Virus Total: https://www.virustotal.com/gui/file-analysis/MmFlYmNlZmRmNTVmODY3ZTg5MTMwMTk3NmNkMmYzNDc6MTU2MzQ4OTY0NQ==/detection
As usual thanks to Skylum for making great software and let me create my tools!
EDIT: After some fine tuning I managed it to reduce my 13,2 GB cached resources folder to 65 MB! I also added a feature which takes care of already created symlinks. This means that you can run it multiple times :)
EDIT2: I updated my tool. It now supports Luminar Flex. Download link updated.
Cheers
Daniel
-
Hi Pete,
when you have already broken links then I think it is the best solution to start with a fresh folder (as you already mentioned)...
Regarding your question: You have to run it from time to time to keep your folder size small. Depending on your editing amount amount I would suggest every 3-6 month.
Cheers
Daniel -
Hi Daniel,
I downloaded your .exe but since I've already moved my photos and broken the links to my Luminar catalog I'm going to just delete the entire Luminar Catalog folder and start over with my photos and edits on a dedicated external HDD.
One question, will your .exe fix the disk bloat problem forever or will I need to run it regularly to keep Luminar from consuming disk space? Do you recommend running Helper after each Luminar session? Weekly? Monthly?
Thanks
-Pete-
-
Thank for fixing the link, however, when I run Analyze Files from the Resource Center, it crashes the app.
************** Exception Text **************
System.OverflowException: Arithmetic operation resulted in an overflow.
at mplumpack.LogInProgressBar.OnPaint(PaintEventArgs e)
at System.Windows.Forms.Control.PaintWithErrorHandling(PaintEventArgs e, Int16 layer)
at System.Windows.Forms.Control.WmPaint(Message& m)
at System.Windows.Forms.Control.WndProc(Message& m)
at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)Running on a Surface Pro 3 with Windows 10 v1903.
-
Hi Daniel,
I used your tool again today and it downsized my Library from around 32GB to 6GB - impressive.
I used the latest version and my Luminar 3 is working perfectly.
Thank you ! it really helps to tame the amount of memory on my SSD Luminar takes normally.
I also like the other things you put in there.
Is there any way we can support you?
All the best,
Robin
-
You've inspired me Daniel.
The SQLite DB for Luminar is easy to navigate, so I feel the need to write an app to skim through the IMAGE_USER_ATTRIBUTES table and generate/update .xmp files. It won't be as flash as having it happen in real time, but at least it will work and users will be able to share ratings/colors between programs.
Here goes...
-
Hi Rob,
thanks for your feedback! I really appreciate this.
Regarding support: I am happy when other people try the software and provide feedback like you and Roy did. This way we can improve Luminar Helper together. If you find any bugs, just let me know ;)
@Rodney: This is a great idea!! If you like I can share the source code with you. I really have to start a github repo...Anyway currently I´m working on the previews especially on the preview folder size. Goal is to have the complete catalog folder at a small size to back it up without using much space. Also reducing size on OS drive.
Cheers
Daniel -
Hi Dayo,
I already shared the sources with the skylum devs ;) I don´t know if they port it to Mac. I´ll also create a github repo once I have time.
Anyway I´m pretty sure this can also be achieved by some kind of script on Macs. "Unfortunately" I´m a Windows guy and I have no clue about Macs...
Cheers
Daniel -
Hi Dayo,
sorry I didn´t get the first post right. Yes sure you can have the sources. I uploaded it to https://www.mediafire.com/file/necf9s03wn2gozp/mplumpack.zip/file
I put some comments into code; hope this helps. Anyway if you have questions regarding the cleaner (or other features) feel free to post them here ;)
Luminar for Windows cannot use Plugins :( That´s why I cannot reproduce this behavior. But I know what you mean; usually it should clean up automatically when the files aren´t needed anymore...
Currently I´m quite busy due to work etc. But one point on my to do list is still the preview folder. I don´t know how exactly it is managed with the Mac version, but under Windows different "kinds" of previews are created:
1. small previews (120 pixels)2, medium previews (300 pixels)
3. big previews (600 pixels)
those are used in library in Luminar. Then there is a another one:
4. even bigger previews (900 pixels) --> This one is used when you open the picture in edit mode.What I figured out is that Luminar does not need some of the previews. For example the biggest one is shown when you want to edit a picture and the picture is still developed/rendered in the background. I also "observed" that Luminar uses higher resolution pictures even when you use small previews. Meaning a 600 pixel image is shown when you use the smallest previews of 120 pixels --> Please excuse my bad English
If you can convert the code (it is VB) it would be awesome! I really hope there is no need to write it from scratch!
Cheers
Daniel -
Hi Daniel,
I did installed the tool today to reduce the catalog folder.
But I get the following error.
See the end of this message for details on invoking
just-in-time (JIT) debugging instead of this dialog box.************** Exception Text **************
System.OverflowException: Arithmetic operation resulted in an overflow.
at mplumpack.LogInProgressBar.OnPaint(PaintEventArgs e)
at System.Windows.Forms.Control.PaintWithErrorHandling(PaintEventArgs e, Int16 layer)
at System.Windows.Forms.Control.WmPaint(Message& m)
at System.Windows.Forms.Control.WndProc(Message& m)
at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)
************** Loaded Assemblies **************
mscorlib
Assembly Version: 4.0.0.0
Win32 Version: 4.8.3815.0 built by: NET48REL1LAST_C
CodeBase: file:///C:/Windows/Microsoft.NET/Framework64/v4.0.30319/mscorlib.dll
----------------------------------------
Luminar Helper
Assembly Version: 1.0.1.0
Win32 Version: 1.0.1.0
CodeBase: file:///C:/Program%20Files/Luminar%20Helper/Luminar%20Helper.exe
----------------------------------------
Microsoft.VisualBasic
Assembly Version: 10.0.0.0
Win32 Version: 14.8.3752.0 built by: NET48REL1
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/Microsoft.VisualBasic/v4.0_10.0.0.0__b03f5f7f11d50a3a/Microsoft.VisualBasic.dll
----------------------------------------
System
Assembly Version: 4.0.0.0
Win32 Version: 4.8.4001.0 built by: NET48REL1LAST_C
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System/v4.0_4.0.0.0__b77a5c561934e089/System.dll
----------------------------------------
System.Core
Assembly Version: 4.0.0.0
Win32 Version: 4.8.4001.0 built by: NET48REL1LAST_C
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System.Core/v4.0_4.0.0.0__b77a5c561934e089/System.Core.dll
----------------------------------------
System.Windows.Forms
Assembly Version: 4.0.0.0
Win32 Version: 4.8.4001.0 built by: NET48REL1LAST_C
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System.Windows.Forms/v4.0_4.0.0.0__b77a5c561934e089/System.Windows.Forms.dll
----------------------------------------
System.Drawing
Assembly Version: 4.0.0.0
Win32 Version: 4.8.3752.0 built by: NET48REL1
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System.Drawing/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Drawing.dll
----------------------------------------
System.Configuration
Assembly Version: 4.0.0.0
Win32 Version: 4.8.3752.0 built by: NET48REL1
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System.Configuration/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Configuration.dll
----------------------------------------
System.Xml
Assembly Version: 4.0.0.0
Win32 Version: 4.8.3752.0 built by: NET48REL1
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System.Xml/v4.0_4.0.0.0__b77a5c561934e089/System.Xml.dll
----------------------------------------
System.Runtime.Remoting
Assembly Version: 4.0.0.0
Win32 Version: 4.8.3752.0 built by: NET48REL1
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System.Runtime.Remoting/v4.0_4.0.0.0__b77a5c561934e089/System.Runtime.Remoting.dll
----------------------------------------
Accessibility
Assembly Version: 4.0.0.0
Win32 Version: 4.8.3752.0 built by: NET48REL1
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/Accessibility/v4.0_4.0.0.0__b03f5f7f11d50a3a/Accessibility.dll
----------------------------------------
System.Xml.Linq
Assembly Version: 4.0.0.0
Win32 Version: 4.8.3752.0 built by: NET48REL1
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System.Xml.Linq/v4.0_4.0.0.0__b77a5c561934e089/System.Xml.Linq.dll
----------------------------------------
System.IO.Compression.FileSystem
Assembly Version: 4.0.0.0
Win32 Version: 4.8.3752.0
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System.IO.Compression.FileSystem/v4.0_4.0.0.0__b77a5c561934e089/System.IO.Compression.FileSystem.dll
----------------------------------------
System.IO.Compression
Assembly Version: 4.0.0.0
Win32 Version: 4.8.3752.0 built by: NET48REL1
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System.IO.Compression/v4.0_4.0.0.0__b77a5c561934e089/System.IO.Compression.dll
----------------------------------------************** JIT Debugging **************
To enable just-in-time (JIT) debugging, the .config file for this
application or computer (machine.config) must have the
jitDebugging value set in the system.windows.forms section.
The application must also be compiled with debugging
enabled.For example:
<configuration>
<system.windows.forms jitDebugging="true" />
</configuration>When JIT debugging is enabled, any unhandled exception
will be sent to the JIT debugger registered on the computer
rather than be handled by this dialog box.Could you tell me, where the issue is?
My catalog is not in the default location. L3 3.1.3, Windows 10 1903
Regards,
Kay-Uwe
-
Just TERRIBLE! I am a casual user and after having to rebuild my library I'm now experiencing over 77% Memory usage in Luminar. I am not a computer geek and do not want to be....I just want my programs to work as they should. I'll try rebuilding as suggested, but dangit Skylum FIX the dang problem on your end!!!!! I am a Win 10 user on Surface Pro 2 with all my photos stored on an external HDD.
-
Here's an update. Deleted all catalogs and reloaded Luminar from scratch. I let it decide where to store stuff. Did some windows updates, cleaned PC with CCleaner including register. Opened Aurora and create plugin to Luminar. Closed everything. Ran Luminar Helper, or tried to and got this error message. I checked and the folder is exactly where Luminar decided to put it.So WHAT THE HELL LUMINAR??!! Oh yeah you're gonna tell me to register the problem outside the forum. GREAT!!
-
Hi Ed,
We regret to hear that you're facing issues. Please, kindly be informed that we plan to release a major performance update pretty soon. We will announce the date a bit later. However, if you want us to help you troubleshoot the issue you're facing further, we recommend you to contact us via email since it will be more convenient for both of us to communicate and keep track of the issue. Thanks for your understanding!
Please sign in to leave a comment.
Comments
27 comments