Camel to file

When Camel is producing files writing files there are a few gotchas affecting how to set a filename of your choice. If such a filename is not desired, then you must provide a filename in the CamelFileName message header. The constant, Exchange. Filename can be set either using the expression option or as a string-based File Language expression in the CamelFileName header.

See the File Language for syntax and samples. Beware if you consume files from a folder where other applications write files directly. Take a look at the different readLock options to see what suits your use cases. The best approach is however to write to another folder and after the write move the file in the drop folder.

The other read lock options rely on Java File API that sadly is not always very good at detecting this. You may also want to look at the doneFileName option, which uses a marker file done to signal when a file is done and ready to be consumed.

If you want only to consume files when a done file exists, then you can use the doneFileName option on the endpoint. Will only consume files from the bar folder, if a file name done exists in the same directory as the target files.

Camel will automatically delete the done file when it's done consuming the files. However its more common to have one done file per target file. This means there is a correlation. To do this you must use dynamic placeholders in the doneFileName option. Currently Camel supports the following two dynamic tokens: file:name and file:name. The consumer only supports the static part of the done file name as either prefix or suffix not both.

In this example only files will be polled if there exists a done file with the name file name. For example. After you have written af file you may want to write an additional done file as a kinda of marker, to indicate to others that the file is finished and has been written.

To do that you can use the doneFileName option on the file producer endpoint. Will for example create a file named done-foo. Will for example create a file named foo. Listen on a directory and create a message for each file dropped there.

Copy the contents to the outputdir and delete the file in the inputdir. Will scan recursively into sub-directories. Will lay out the files in the same directory structure in the outputdir as the inputdir , including any sub-directories. If you want to store the files in the outputdir directory in the same directory, disregarding the source directory layout e.

Camel will by default move any processed file into a. The body will be a File object that points to the file that was just dropped into the inputdir directory.

By default the file endpoint sends a FileMessage which contains a File object as the body. By converting the File to a String , the message will contain the file content which is probably what you want. Camel is of course also able to write files, i. In the sample below we receive some reports on the SEDA queue that we process before they are written to a directory.

Using a single route, it is possible to write a file to any number of subdirectories. If you have a route setup as such:. In this sample we want to move consumed files to a backup folder using today's date as a sub-folder name:.

See File Language for more samples. Camel supports Idempotent Consumer directly within the component so it will skip already processed files.

By default Camel uses a in memory based store for keeping track of consumed files, it uses a least recently used cache holding up to entries. You can plugin your own implementation of this store by using the idempotentRepository option using the sign in the value to indicate it's a referring to a bean in the Registry with the specified id.

In this section we will use the file based idempotent repository org. FileIdempotentRepository instead of the in-memory based that is used as default. This repository uses a 1st level cache to avoid reading the file repository. It will only use the file repository to store the content of the 1st level cache. Thereby the repository can survive server restarts. It will load the content of the file into the 1st level cache upon startup. The file structure is very simple as it stores the key in separate lines in the file.

By default, the file store has a size limit of 1mb. When the file grows larger Camel will truncate the file store, rebuilding the content by flushing the 1st level cache into a fresh empty file.

We configure our repository using Spring XML creating our file idempotent repository and define our file consumer to use our repository with the idempotentRepository using sign to indicate Registry lookup:. In this section we will use the JPA based idempotent repository instead of the in-memory based that is used as default.

MessageProcessed as model. And yes then we just need to refer to the jpaStore bean in the file consumer endpoint using the idempotentRepository using the syntax option:. Camel supports pluggable filtering strategies. You can then configure the endpoint with such a filter to skip certain files being processed. In the sample we have built our own filter that skips files starting with skip in the filename:.

And then we can configure our route using the filter attribute to reference our filter using notation that we have defined in the spring XML file:. The ANT path matcher is shipped out-of-the-box in the camel-spring jar. So you need to depend on camel-spring if you are using Maven. The reasons is that we leverage Spring's AntPathMatcher to do the actual matching. Camel supports pluggable sorting strategies. This strategy it to use the build in java.

Comparator in Java. You can then configure the endpoint with such a comparator and have Camel sort the files before being processed. And then we can configure our route using the sorter option to reference to our sorter mySorter we have defined in the spring XML file:.

This strategy it to use the File Language to configure the sorting. The sortBy option is configured as follows:. Where each group is separated with semi colon. In the simple situations you just use one group, so a simple example could be:.

This will sort by file name, you can reverse the order by prefixing reverse: to the group, so the sorting is now Z.. As we have the full power of File Language we can use some of the other parameters, so if we want to sort by file size we do:.

You can configure to ignore the case, using ignoreCase: for string comparison, so if you want to use file name sorting but to ignore the case then we do:. And then we want to group by name as a 2nd option so files with same modifcation is sorted by name:. Now there is an issue here, can you spot it? Well the modified timestamp of the file is too fine as it will be in milliseconds, but what if we want to sort by date only and then subgroup by name?

Well as we have the true power of File Language we can use the its date command that supports patterns. So this can be solved as:. Yeah, that is pretty powerful, oh by the way you can also use reverse per group, so we could reverse the file names:.

The option processStrategy can be used to use a custom GenericFileProcessStrategy that allows you to implement your own begin , commit and rollback logic. For instance lets assume a system writes a file in a folder you should consume. But you should not start consuming the file before another ready file has been written as well. The filter option allows you to implement a custom filter in Java code by implementing the org. GenericFileFilter interface. This interface has an accept method that returns a boolean.

Return true to include the file, and false to skip the file. From Camel 2. Add a comment. Active Oldest Votes. Improve this answer. Laurent Chabot Laurent Chabot 2 2 silver badges 8 8 bronze badges. Sign up or log in Sign up using Google. Sign up using Facebook. Sign up using Email and Password.

Post as a guest Name. Email Required, but never shown. The Overflow Blog. Podcast Making Agile work for data science. Stack Gives Back Featured on Meta. New post summary designs on greatest hits now, everywhere else eventually.

Related 0. Configures the number of permitted calls when the CircuitBreaker is half open. The size must be greater than 0. Default size is Configures the size of the sliding window which is used to record the outcome of calls when the CircuitBreaker is closed. Sliding window can either be count-based or time-based.

The slidingWindowSize must be greater than 0. The minimumNumberOfCalls must be greater than 0. Default slidingWindowSize is Configures the type of the sliding window which is used to record the outcome of calls when the CircuitBreaker is closed. Configures the duration threshold seconds above which calls are considered as slow and increase the slow calls percentage. Default value is 60 seconds.

Configures a threshold in percentage. The CircuitBreaker considers a call as slow when the call duration is greater than slowCallDurationThreshold Duration. When the percentage of slow calls is equal or greater the threshold, the CircuitBreaker transitions to open and starts short-circuiting calls. Default value is percentage which means that all recorded calls must be slower than slowCallDurationThreshold. Configures the wait duration in seconds which specifies how long the CircuitBreaker should stay open, before it switches to half open.

Enables writable stack traces. When set to false, Exception. This may be used to reduce log spam when the circuit breaker is open as the cause of the exceptions is already known the circuit breaker is short-circuiting calls.

RestApiProcessorFactory is registered in the registry. If either one is found, then that is being used. This can be used when using components such as camel-servlet where the deployed web application is deployed using a context-path. The route will by default use an auto assigned route id. To use an specific hostname for the API documentation eg swagger This can be used to override the generated host with this configured hostname. Allows to configure as many additional properties for the api documentation swagger.

For example set property api. Whether vendor extension is enabled in the Rest APIs. If enabled then Camel will include additional information as vendor extension eg keys starting with x- such as route ids, class names etc. This can be turned on, to enable this check. The default value is false.

If no component has been explicit configured, then Camel will lookup if there is a Camel component that integrates with the Rest DSL, or if a org. RestConsumerFactory is registered in the registry. Allows to configure as many additional properties for the rest component in use.

Allows to configure as many additional properties for the rest consumer in use. Sets a leading context-path the REST services will be using.

Or for components such as camel-jetty or camel-netty-http that includes a HTTP server. Allows to configure as many additional properties for the data formats in use.

For example set property prettyPrint to true to have json outputted in pretty mode. The prefixes are: json. For example a key with value xml. A key without a prefix is a common key for all situations. Allows to configure as many additional properties for the rest endpoint in use. If no hostname has been explicit configured, then this resolver is used to compute the hostname the REST service will be using. Name of specific json data format to use. By default json-jackson will be used.

Important: This option is only for setting a custom name of the data format, not to refer to an existing data format instance. The port number to use for exposing the REST service. Notice if you use servlet component then the port number configured here does not apply, as the port number in use is the actual port number the servlet component is using.

Though in those situations setting the port number here, allows tooling and JMX to know the port number, so its recommended to set the port number to the number that the servlet engine uses. Sets the location of the api document swagger api the REST producer will use to validate the REST uri and query parameters are valid accordingly to the api document.

This requires adding camel-swagger-java to the classpath, and any miss configuration will let Camel fail on startup and report the error s. The location of the api document is loaded from classpath by default, but you can use file: or http: to refer to resources to load from file or http url. The scheme to use for exposing the REST service.

Usually http or https is supported. The default value is http. Whether to skip binding on output if there is a custom HTTP error code header. Whether to use X-Forward headers for Host and related setting. The default value is true. Name of specific XML data format to use. By default jaxb will be used. The pattern name refers to the CamelContext name, to match on the current CamelContext only.

If enabled it allows to discover these contexts, if false then only the current CamelContext is in use.

Edit this Page. File Since Camel 1. The file language is merged with Simple language which means you can use all the file syntax directly within the simple language. File Language options The File language supports 2 options, which are listed below.

Sets the class name of the result type type from output. Syntax This language is an extension to the Simple language so the Simple syntax applies also. File token example Relative paths We have a java. Absolute paths We have a java. Samples You can enter a fixed file name such as myfile. Dependencies The File language is part of camel-core.