To enable it, enable compat-v0_3
feature.
The built-in compatibility layer emulates what v0.3.x used to behave, including the serialization, deserialization, number encoding and number decoding.
With this emulation, you can migrate your code with less pain–
most of the current components will mostly behave as before. However,
all of them are marked as deprecated
, and we may remove
this layer in the future major version (such as 0.5.x).
DO NOT DEPEND ON THIS LAYER, and migrate your codebase as long as possible! This layer may slow down your program, and the layer may have some unexpected changes compared to the 0.3.x version (for example, the Error variant).
Assuming your code is using encode_bbo
(or encode_orderbook
, whatever):
let encoded: Vec<u8> = encode_bbo(your_bbo_msg).unwrap();
First, you have to convert your Msg
in crypto-crawler,
to libstock's Structure
. You can approach it with try_into()
:
// OrderBookMsg → OrderbookStructure, KlineMsg → KlineStructure,
// TradeMsg → TradeStructure, FundingRateMsg → FundingRateStructure,
// so on.
let bbo_structure = BboStructure::try_from(your_bbo_msg)?
Our new serialization method accepts any type implementing Write
–
in other words, you can pass your Socket, File and anything else writable
to the serialize()
method:
// Use our serialization method.
use wmjtyd_libstock::data::serializer::StructureSerializer;
bbo_structure.serialize(your_file)?;
If you prefer the current way, you need to manage your own buffer
(however, you can pass any buffer implementing Write
for your case.)
let mut buffer = Vec::new();
bbo_structure.serialize(buffer)?;
Assuming your code is using decode_bbo
(or decode_orderbook
, whatever):
let decoded: BboMsg = decode_bbo(src).unwrap();
First, use our deserialization method to construct the Structure.
It accepts any type implementing Read
trait, so you can just
pass your socket, file or anything else readable to the deserialize()
method.
// Use our deserialization method.
use wmjtyd_libstock::data::serializer::StructureDeserializer;
let decoded = BboStructure::deserialize(&mut your_data_source);
Note that if your data source is a Vec<T>
, you must turn it to a mutable-immutable slice.
It is counterintuitive, but it works ;)
let decoded = BboStructure::deserialize(&mut your_vec.as_slice());
Finally, convert your Structure back to the crypto-crawler's Msg:
let msg = BboMsg::try_from(decoded)?;
We have rewritten the encoding and decoding methods for clearer architecture, robust and performance. However, it introduces some breaking change though it can be easily resolved.
Assuming you have such this code:
let num = 1234_5678.to_string();
u32::encode_bytes(&num)
First, You need to convert your integer, float number, or number string
to rust_decimal::Decimal
. We have re-exported it in data::num
so you won't need to add another dependencies by yourself.
Take the above as an example, we can rewrite it to:
// Integer
let num = rust_decimal::from(1234_5678);
// Float; you should do your own error handling.
let num = rust_decimal::from_f32(1234.5678).map_err(...)?;
// Number string; you should do your own error handling.
let num = rust_decimal::from_str_exact("1234.5678")?;
After that, we include our new encode API and encode the num
:
use wmjtyd_libstock::data::num::Encoder;
let num_bytes: [u8; 5] = num.encode()?;
To decode the number, use Decoder::decode
:
use wmjtyd_libstock::data::num::Decoder;
let num = Decoder::decode(&num_bytes)?;
We did not write the compatibility layer for fields, as it used to be internally used in the serialization and deserialization, and most people may not operate with this.
Some notable big changes to migrate your codebase:
data::types
has merged intodata::fields
.- All the
-Repr
are renamed to-Field
. try_from_reader
→deserialize_from_reader
from_bytes
(ortry_from_bytes
) →deserialize
to_bytes
(ortry_to_bytes
) →serialize
SymbolPairField
is no longer a tuple – its fields has been named now.ReadExt
has been merged toDeserializer
.
Though we reshaped and rewrote all the message modules, there are not really big changes for users.
Migrate nanomsg
writers
Assuming you used to write this way:
use std::io::Write;
use wmjtyd_libstock::message::nanomsg::{Nanomsg, NanomsgProtocol};
let nanomsg = Nanomsg::new("ipc:///tmp/cl-nanomsg-old-api-w.ipc", NanomsgProtocol::Pub);
if let Ok(mut nanomsg) = nanomsg {
nanomsg.write_all(b"Hello World!").ok();
}
First, update your Nanomsg
imports to NanomsgPublisher
.
We have separated Nanomsg
to NanomsgPublisher
and NanomsgSubscriber
,
which implementing the corresponding Publisher
and Subscriber
:
use wmjtyd_libstock::message::nanomsg::NanomsgPublisher;
And then, construct your NanomsgPublisher
. Note that no any
additional parameter need to pass in this construction. For address,
we will bind our address in another function later.
let nanomsg = NanomsgPublisher::new();
new()
returns Result
, so error handling is needed here.
After that, let's bind our address to Nanomsg:
use wmjtyd_libstock::message::traits::Bind;
if let Ok(mut nanomsg) = nanomsg {
nanomsg.bind("ipc:///tmp/cl-nanomsg-new-api-w.ipc").ok();
Note that we includes traits::Bind
here. To use the bind
method, you must introduce our Bind
trait here. It is
necessary for our high-level abstraction. Finally, we introduce
Write
trait, just like what we did before.
use std::io::Write; // or use wmjtyd_libstock::message::traits::Write;
nanomsg.write_all(b"Hello World!").ok();
}
Besides, we provide the async API of nanomsg
now!
use tokio::io::AsyncWriteExt; // or use wmjtyd_libstock::message::traits::AsyncWriteExt;
nanomsg.write_all(b"Hello World!").await.ok();
}
The full synchronous example:
use wmjtyd_libstock::message::nanomsg::NanomsgPublisher;
use wmjtyd_libstock::message::traits::{Bind, Write};
let nanomsg = NanomsgPublisher::new();
if let Ok(mut nanomsg) = nanomsg {
nanomsg.bind("ipc:///tmp/cl-nanomsg-new-api-w.ipc").ok();
nanomsg.write_all(b"Hello World!").ok();
}
Migrate zeromq
writers
Just like nanomsg
, but use zeromq::ZeromqPublisher
. Asynchronous Example:
use wmjtyd_libstock::message::traits::{AsyncWriteExt, Bind};
use wmjtyd_libstock::message::zeromq::ZeromqPublisher;
let zeromq = ZeromqPublisher::new();
if let Ok(mut zeromq) = zeromq {
zeromq.bind("ipc:///tmp/cl-zeromq-new-api-w-a.ipc").ok();
zeromq.write_all(b"Hello World!").await.ok();
}
Migrate nanomsg
readers
WIP. Really similar to what we did in writers. Example of this new synchronous API:
use wmjtyd_libstock::message::nanomsg::NanomsgSubscriber;
use wmjtyd_libstock::message::traits::{Connect, Read, Subscribe};
let nanomsg = NanomsgSubscriber::new();
if let Ok(mut nanomsg) = nanomsg {
nanomsg.connect("ipc:///tmp/cl-nanomsg-new-api-r.ipc").ok();
nanomsg.subscribe(b"").ok();
let mut buf = [0; 12];
nanomsg.read_exact(&mut buf).ok();
assert_eq!(b"Hello World!", &buf);
}
Migrate zeromq
readers
WIP. All of the APIs are followed our new rule and thus the logic is just the same. Example of this new asynchronous API:
use wmjtyd_libstock::message::zeromq::ZeromqSubscriber;
use wmjtyd_libstock::message::traits::{Connect, AsyncReadExt, Subscribe};
let zeromq = ZeromqSubscriber::new();
if let Ok(mut zeromq) = zeromq {
zeromq.connect("ipc:///tmp/cl-zeromq-new-api-r.ipc").ok();
zeromq.subscribe(b"").ok();
let mut buf = [0; 12];
zeromq.read_exact(&mut buf).await.ok();
assert_eq!(b"Hello World!", &buf);
}
message
has been largely rewritten.- Methods under
data
module are mostly reshaped. You may need to rewrite your code to adapt this new architectiure.- We have abstracted a
Structure
– To serialize yourBboMsg
, do.try_into()
first.
- We have abstracted a
- Added a End-Of-Data flag (
\0
) to all the current structure.
- Rewrite
data
module for robust and performance. - Implement
Eq
forfile/writer
. - Implement a general serializer and deserializer for field and structure.
StructSerializer
: We now accept any type implementedstd::io::Write
, instead ofBufWriter
only!StructDeserializer
: We now accept any type implementedstd::io::Read
, instead ofBufReader
only!
- Add the ability to convert other structures other than what
crypto-crawler
provides! 🎉- Just implement the
TryFrom
between your structure and our structure. - Our incoming C binding adds the ability to operate with
libstock
's serialization and deserialization feature in C/C++. - Our work-in-progress Java binding also adds such the ability like C/C++'s.
- Just implement the
- Derive more useful traits for our structures, such as
Debug
,PartialEq
,Eq
,Hash
,Clone
.- Also our fields ;)
- Update dependencies.
- WIP: merge crypto-crawler changes to upstream
- Automatically abort the old CI tasks by
setting the
concurrent
flag. - Set
rust-toolchain.toml
to beta toolchain.
file/reader
:open()
returnsResult
instead ofOption
now.- For better backtrace
file/writer
: Set the size of the length field of written data structure to 2 bytes.- It used to be 'usize'. It is an implementation mistake.
file/hex
: removed two deprecated methodsfile/reader
: move path to<date>/<name>.csv
- Updated the fields of
data
. file/datadir
: need to pass a timestamp
slack
: For sending notifications to Slack with Slack Hook.message
: A basic encap ofnanomsg
andzeromq
for subscribing and publishing.- (beta 2) implement DerefMut to
Socket
formessage
- (beta 2) implement DerefMut to
message
: AddSubscribe
trait for generalizesubscribe()
method.zeromq
andnanomsg
both have implemented this.
file/hex
: Add the encode/decode capability for negative numbers- WIP: currently dirty but works.
message/zeromq
: migrate to the ZeroMQ implemented in Rust.
file/reader
: Makeread()
returns the exact data- Currently, it always returns
[]
due to an implementation mistake.
- Currently, it always returns
file/writer
: Flush buffer after the data written ended.- (beta 2)
message
: useconnect
instead ofbind
for Sub data/*/decode
:ReceivedTimestampRepr
should be 6 bytes instead of 8 bytes.
- Upgrade
crypto-crawler-rs
to92aee0d37e228e53dd994a17058a7f819e005446
- Clean up the unnecessary dependencies.
- Deprecated
file/ident
.
- Add encode/decode tests to
data/bbo
- WIP: add other encode/decode tests to
data/*
!
- WIP: add other encode/decode tests to
file/reader
: addinfo
logger on new() for better debugging
- Install
nanomsg
formessage
test. - Introduce
cargo nextest
for faster build.
Symble
is now renamed toSymbol
.Symbol
field is now serialized as the big endian representation ofu8
.- Map
EXCHANGE
andINFOTYPE
are replaced to the enumerationExchange
andInfoType
.- Use
strum
to provide the map-like function.
- Use
- Some methods in
data::hex
are removed.- Replace your current code to the native method (
.to_be_bytes()
) - (beta 5) For UNIX timestamp, use
unix_ms_to_six_byte_hex
andsix_byte_hex_to_unix_ms
instead.
- Replace your current code to the native method (
- Removed and added some new unused error variants.
data::orderbook
,data::trade
: UseBufReader
instead of our based-on-Atomic seek reader.data::orderbook
,data::trade
: Usedata::fields
to serialize & deserialize.- Separate the serialization & deserialization of fields to
data::fields
- Replace the usize converter of
hex::encode_num_to_bytes
to a robust implementation- Related:
Some methods in data::hex are removed.
- Related:
data::types
: AddExchange
andInfoType
while removingEXCHANGE
andINFOTYPE
.data::orderbook
: Improve the logic ofgenerate_diff
.data::hex
: Deprecate old inextensible methods (encode_num_to_bytes
&decode_num_to_bytes
)- use
u32::encode_bytes
andu32::decode_bytes
for 5-byte encoding & decoding
- use
data::hex
: Add the u64 to 10-byte encode & decode support- use
u64::encode_bytes
andu64::decode_bytes
for 10-byte encoding & decoding
- use
data::types
: Add more exchanges, and addPERIOD
.- Add the encoder and decoder of
bbo
andkline
. file::writer
: Addtracing
log.file::writer
: Refactored for extensibility.- Change to wmjtyd's
crypto-crawler
andcrypto-msg-parser
. - (beta 5) Add methods to serialize and deserialize UNIX timestamp to 6-byte
u64
.
- Also test if
cargo doc
can generate correct documentation. - Run
cargo clippy
andcargo fmt --check
- Add a production example of
file::writer
. - (beta 4)
data::hex
add the testcase ofi8
→u8
andu8
→i8
file::writer
: Don't create the directory if it has been existed.file::writer
:.stop()
can't work properly.- (beta 4)
{InfoTypeExpr,ExchangeTypeRepr}::try_from_str
should be from lowercase- Added the test case of this.
- (beta 5) UNIX timestamp should serialize and deserialize to 6-byte
u64
.