Database VS File System - images

ACDC

Way back in time, awareim only had the option of storing Document images in the DB , then they updated the feature with a file system option. I knew sometime in the future we would have to convert the application to store images in the filesystem , because of the sheer volume of images coming into the system and my gut feeling the DB was not a good option.

Well it finally happened , the wheels fell off – Suddenly when trying to publish a new update to the production system AwareIM could not publish because the time taken to publish was longer than the expected time Aware allows you to publish.. huh , this was news to me – after frantically get hold of support, I learnt that the AwareIM server will timeout (go away) if it takes longer than a certain time frame to make the changes while in the publishing procedure. This results in server failure and reboot needed, not to mention database corruption and restoring from huge back-up files

To cut a long story short, the culprit was the images in the database (changes to the tables where image objects were stored were taking ages to complete ). So we were forced to now move all the images into the file system - this was a complicated task, but we eventually got there by creating a small app in aware that exported all the images and later made some changes in the main app re-assigning the current document attribute to “File System”

In hindsight we should have done this long time ago, the applications database is now 90 percent smaller, requires a lot less ram to be efficient and takes 4 minutes to back up as opposed to 90 minutes. Publishing to the production system is now a lot more faster and efficient and best of all the app is now a lot faster (like its running on steroids) Oh also, maybe we can get real-time replication working now – everything seems so simple now !!!!!!!!

So the moral of the story, always use file system option, otherwise, it will come back to bite you

BTW, if you need to go through the same process of converting images from the DB to the File system I can help with some good pointers to get it done very efficiently if you are interested

RentProperty

Hi ACDC

We went through this Painful process with all our older applications too and I can vouch that It's definitely the way to go. There is one problem however and that is that Image Attributes themselves cannot be put in a file system. You can only store images in the file system if it is a document attribute... And document attributes cannot be displayed as Images in queries.

Hoping that aware Will eventually implement File System option for Image Attributes as well.

Kind Regards

ACDC

Hoping that aware Will eventually implement File System option for Image Attributes as we

Hi Hein
I wonder why there is no File System option as Picture is stored in the DB in the same way as Document. I did not realise that. From what I know now I would strongly urge support to consider this. If we used picture attributes in the same way as we use document attributes we would be in deep crap right now.

btw, if we had to use Picture attributes I would store them as a jpg or png Document (file system) and then reference them in an html panel as opposed using a Picture attribute

RentProperty

I know, I would also think that it should work exactly the same but that option is not available for Picture attributes. We ended up developing custom plugin to store the photos in S3 and return the URL and then delete the Actual Photo from the Picture attribute and just display the URL.... But it's a pain to get that working. Would be so much easier to just store it in a file system on the server same way as documents.

Anywayz... Hopefully after V8.1 onward can focus on things like more flexible REST services and optimization of things like the use of pictures/file systems etc, and auto cleanup of temp files etc. 🙂 Wish there was 10 Vlads so everything can happen at once. 🙂 haha
But at least we know the guys at Aware are doing their best to continually improve so hopefully these things are somewhere in the pipeline going forward. I'm super keen for V8 and V8.1.

BenHayat

Yes Tony, I did some experimentation a few months ago and I [think] I created a thread related to the results. Basically, to stay away from storing documents or images (unless it a 64x64 thumbnail) in the database.
However, for large # of Blobs (say PDFs or DOCs or Images), you need to build an extensive "Document Management System" to manage (add, move, delete) documents in different folders and making sure you have a solid way to know it's location and never lose that link.
If you do it this way, then your Aware (especially during user's start up) runs much faster.

intra

Hi Guys,

Keen to know at what size (row wise) did you start to run into problems?

What was the avg file size of the document in the attribute?

BenHayat

Part of my app as a SaaS app, ALL users have access to some Common images which are now stored in DB. Although the number of images and sizes are not large, but the the sequence of actions that Aware has to perform to get image from DB to user's browser multiple bu the number of concurrent users, can take a toll on Aware's processing time and memory.

So, I've decided to convert all of my DB images (public or private) to file system and then build an image management system to store them in different folders. I'm thinking even the thumbnails to be stored in file system and not put any burden on Aware server for image conversion.

Any thoughts?

BenHayat

Well guys, I got bad news as to using FILE SYSTEM to store images. Firstly, it's more work managing folders, unique Naming of images, creating thumbnails and updating images. But that's not the bad news, it's just part of being a developer.

The BAD news is that, the images do not show on forms, grids and queries. You only see the name showing. Aware does not grab the image from the stored location and pass it to Tomcat to send it to browser. This is a huge problem, because we need to show them on forms to users and also on grid as a sequence of steps to do tasks (for training).

Going back to DB, we're going to have the same problem ACDC had with volume.

WHAT DO WE DO???

ACDC

Well guys, I got bad news as to using FILE SYSTEM to store images. Firstly, it's more work managing folders, unique Naming of images,

To simplify this, I default all the file names on Disk to the ObjectID where the attribute resides. If there are more than one document on the same object I take care of this by adding a reference onto the object ID to make it unique. So the file names on disk are only relevant to awareim , information as to the the original filename is stored on the object using FILE_NAME function which is exposed to the user, the filename on disk is meanless to the user

As to the Folder setting, I set this in the RegularUser Object or System settings and then embed the folder name in the path of the Attribute filesystem location. eg 😮bject.DocumentPath ='<<LoggedInRegularUser.Path_ExamDocs>><<Exam.ID>>.pdf'

The BAD news is that, the images do not show on forms, grids and queries.

The solution here is to convince support to add the same "File System" option functionality as in Document Type to PICTURE type. This way all the features of the Picture type would be accessible -

BenHayat

Thanks Tony;

I had done pretty much the same way as you did. Except I used GUID for file names. But I spread the images to different folders because of their nature and all those paths are stored in SystemSettings.

I had made the request to Support and it's on ToDo list for 8.1
Let's hope it makes it or it will break my app. 🙁

ACDC

Excellent, can't wait I have a big project coming with Images

BenHayat

ACDC wrote
Excellent, can't wait I have a big project coming with Images

One question ACDC;

When you converted your imaging system from DB to FILE SYSTEM, I assume you either set the DB attribute for image to UNDEFINED or deleted the attribute from the schema.

And as you mentioned this conversion made you Database size a lot smaller.

My question: After nulling or deleting the image attribute, and letting Aware update Database, did the database shrink at this point or did you have to export your data, create an empty DB and re-import data to get a smaller database?

ACDC

Ben,

My question: After nulling or deleting the image attribute, and letting Aware update Database, did the database shrink at this point or did you have to export your data, create an empty DB and re-import data to get a smaller database?

The xxxx_DOC_DATA Attribute is where the image is stored. I deleted this column from the table using a database editor and then recreated the attribute with the editor. This was the LAST step of the process which resulted in the database size reduction. So this attribute remains unused and always null. (You may not have to recreate it, but I chose to do this not knowing how AwareIM would react)

I recommend the following process (changing the existing document attribute pointing to the DB to File System):

Add the File Path Attribute to the object containing the document. (Do not set the document attribute to file system at this stage.)
Create a rule on the object to create and store the PATH to the file
Create a temporary Rule on the object to EXPORT Document to File Path
Temporarily disable all rules on the object except for the above two rules
Create a Process: FIND ALL Object; UPDATE Object - run process (you may want to run the Path Rule and Export rule separately by disabling one of them )
Check if all images were exported correctly and path created (using the Object ID in the filename of the image will assist you to validate the export
Edit the BSV enabling all the rules again and removing the Export Document rule
Change the existing Document attribute to File System and Publish
Check Functionality, the document attribute should be accessing the file system
Delete the xxx_Doc_Data in the DB using a DB editor removing the images from the database
To maintain a real-time backup of the file system images run a peer to peer shared folder setup – you can pm me for more details on this if needed

hope that makes sense

BenHayat

That's what I'm talking about Tony. 😃
I had given it some thought, but not to the extend you laid out.

Depending on when 8.1 comes out and when I go to production. But as of now, I'll be building all FILE SYSTEM features into my app along side with the DB images and when I decide to go live I can choose which one to start with. If I start with DB Images, then I have to go through the above conversion when Support give us FILE SYSTEM with full display features.

Thank you so much!
Hope your foot feels better now, with that expensive medicine. :mrgreen:

ACDC

Ben

I'll be building all FILE SYSTEM features into my app along side with the DB images and when I decide to go live I can choose which one to start with

Yes, that would be a good idea. In the absence of the file system option on the PICTURE attribute, it makes sense to create the Path Attribute now, and also export the image to the file system using a rule on the object. So when a new object is created the image is stored in the DB and the File System. Then when v8.1 arrives just set the attribute to file system and remove the DB image column from the DB. That's if Vlad makes it work the same way as DOCUMENT file system option - maybe you should confirm the same method with him so there are no surprises because the PICTURE attribute has a lot more going on than the DOCUMENT attribute

BenHayat

ACDC wrote
So when a new object is created the image is stored in the DB and the File System. Then when v8.1 arrives just set the attribute to file system and remove the DB image column from the DB.

Brilliant idea. to have both system running side by side.
After 8 is out, I'll discuss this more with Vlad.

BenHayat

Thank you Tony for all your input and feedback!

RentProperty

Would be great if it was flexible enough out of the box to easily allow integration with something like Amazon S3 & Cloud Front. 🙂 Then you would be able to store images either on the server in a file system or in S3 and either way Aware would then know to download and display the images client side based on the image URL instead of getting them from the DB. This takes HUGE load of Tomcat. The image widgets would have to change a bit to accommodate this I assume. As currently you have to do a manual HTML cell implementation if you want to display a URL image.

Looking forward to some really awesome and long awaited new features and improvements in 8.1.

Jaymer

guys, i'm in a bind and i hope one of you will take time to help.

i have a small 60k image that has been stored in a document.
that document is STORED IN FILE SYSTEM.
True, I cannot see the image in a query or form, and I have to click on it in a form for it to appear.
All thats OK.

But I need to print the simple image in its own report, as these 3 reports make up a dossier thats gets emailed as 3 attachments.
I already print an Invoice, and a 2nd document with photos. These Reports already work and are sent to PDF.
When I made a 3rd report for this simple image (because it doesn't exist on all Invoices), all that appears in the detail band is my filename. No image.

Is this the issue y'all refer to in this thread, that only Pictures get imaged by aware?
How am I supposed to get this image to show up on a report? (which ultimately gets saved to PDF, and attached)?

thx

PS _ What good is having the document outside the db if I can't do anything with it except manually click on it to view?