Skip to content

Latest commit

 

History

History
143 lines (96 loc) · 6.57 KB

README.adoc

File metadata and controls

143 lines (96 loc) · 6.57 KB
Raw

FTP Source

Messages emitted by the source are provided as a byte array by default. However, this can be customized using the --mode option:

  • ref Provides a java.io.File reference

  • lines Will split files line-by-line and emit a new message for each line

  • contents The default. Provides the contents of a file as a byte array

When using --mode=lines, you can also provide the additional option --withMarkers=true. If set to true, the underlying FileSplitter will emit additional start-of-file and end-of-file marker messages before and after the actual data. The payload of these 2 additional marker messages is of type FileSplitter.FileMarker. The option withMarkers defaults to false if not explicitly set.

See also MetadataStore options for possible shared persistent store configuration used to prevent duplicate messages on restart.

Input

N/A (Fetches files from an FTP server).

Output

mode = contents

Headers:

  • Content-Type: application/octet-stream

  • file_originalFile: <java.io.File>

  • file_name: <file name>

Payload:

A byte[] filled with the file contents.

mode = lines

Headers:

  • Content-Type: text/plain

  • file_orginalFile: <java.io.File>

  • file_name: <file name>

  • correlationId: <UUID> (same for each line)

  • sequenceNumber: <n>

  • sequenceSize: 0 (number of lines is not know until the file is read)

Payload:

A String for each line.

The first line is optionally preceded by a message with a START marker payload. The last line is optionally followed by a message with an END marker payload.

Marker presence and format are determined by the with-markers and markers-json properties.

mode = ref

Headers:

None.

Payload:

A java.io.File object.

Options

The ftp source has the following options:

Properties grouped by prefix:

file.consumer

markers-json

When 'fileMarkers == true', specify if they should be produced as FileSplitter.FileMarker objects or JSON. (Boolean, default: true)

mode

The FileReadingMode to use for file reading sources. Values are 'ref' - The File object, 'lines' - a message per line, or 'contents' - the contents as bytes. (FileReadingMode, default: <none>, possible values: ref,lines,contents)

with-markers

Set to true to emit start of file/end of file marker messages before/after the data. Only valid with FileReadingMode 'lines'. (Boolean, default: <none>)

ftp.factory

cache-sessions

Cache sessions. (Boolean, default: <none>)

client-mode

The client mode to use for the FTP session. (ClientMode, default: <none>, possible values: ACTIVE,PASSIVE)

host

The host name of the server. (String, default: localhost)

password

The password to use to connect to the server. (String, default: <none>)

port

The port of the server. (Integer, default: 21)

username

The username to use to connect to the server. (String, default: <none>)

ftp.supplier

auto-create-local-dir

Set to true to create the local directory if it does not exist. (Boolean, default: true)

delay-when-empty

Duration of delay when no new files are detected. (Duration, default: 1s)

delete-remote-files

Set to true to delete remote files after successful transfer. (Boolean, default: false)

filename-pattern

A filter pattern to match the names of files to transfer. (String, default: <none>)

filename-regex

A filter regex pattern to match the names of files to transfer. (Pattern, default: <none>)

local-dir

The local directory to use for file transfers. (File, default: <none>)

preserve-timestamp

Set to true to preserve the original timestamp. (Boolean, default: true)

remote-dir

The remote FTP directory. (String, default: /)

remote-file-separator

The remote file separator. (String, default: /)

tmp-file-suffix

The suffix to use while the transfer is in progress. (String, default: .tmp)

metadata.store.dynamo-db

create-delay

Delay between create table retries. (Integer, default: 1)

create-retries

Retry number for create table request. (Integer, default: 25)

read-capacity

Read capacity on the table. (Long, default: 1)

table

Table name for metadata. (String, default: <none>)

time-to-live

TTL for table entries. (Integer, default: <none>)

write-capacity

Write capacity on the table. (Long, default: 1)

metadata.store.jdbc

region

Unique grouping identifier for messages persisted with this store. (String, default: DEFAULT)

table-prefix

Prefix for the custom table name. (String, default: <none>)

metadata.store.mongo-db

collection

MongoDB collection name for metadata. (String, default: metadataStore)

metadata.store.redis

key

Redis key for metadata. (String, default: <none>)

metadata.store

type

Indicates the type of metadata store to configure (default is 'memory'). You must include the corresponding Spring Integration dependency to use a persistent store. (StoreType, default: <none>, possible values: mongodb,redis,dynamodb,jdbc,zookeeper,hazelcast,memory)

metadata.store.zookeeper

connect-string

Zookeeper connect string in form HOST:PORT. (String, default: 127.0.0.1:2181)

encoding

Encoding to use when storing data in Zookeeper. (Charset, default: UTF-8)

retry-interval

Retry interval for Zookeeper operations in milliseconds. (Integer, default: 1000)

root

Root node - store entries are children of this node. (String, default: /SpringIntegration-MetadataStore)

Examples

java -jar ftp_source.jar --ftp.supplier.remote-dir=foo --file.consumer.mode=lines --ftp.factory.host=ftpserver \
         --ftp.factory.username=user --ftp.factory.password=pw --ftp.local-dir=/foo