Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[C++] Creating a buffer directly into a memory mapped file without leading unused bytes #7760

Closed
ridilculous opened this issue Jan 6, 2023 · 4 comments
Labels

Comments

@ridilculous
Copy link

I'm trying out the suggestion 3 from here and am wondering if there's a way to get rid of the leading unused bytes this results in without manually moving the buffer to the front and truncating the file afterwards.

From my understanding, the only way to do this would be to know the final size in advance which isn't possible with optional fields.
Am i missing something? How do people usually do this?
I guess only having item 9 from here would enable this?

Code for reference:

class MmapAlloc : public ::flatbuffers::Allocator {
    boost::iostreams::mapped_file& mmapFile_;
public:
    MmapAlloc(boost::iostreams::mapped_file& mmapFile) : mmapFile_(mmapFile) {}
    uint8_t* allocate(size_t size) override { return (uint8_t*)mmapFile_.data(); }
    void deallocate(uint8_t* p, size_t size) override {}
    uint8_t* reallocate_downward(uint8_t* old_p, size_t old_size, size_t new_size, size_t in_use_back, size_t in_use_front) override {
        throw std::runtime_error("not allowed");
    }
};

boost::iostreams::mapped_file_params params;
params.path = "test";
params.flags = boost::iostreams::mapped_file::mapmode::readwrite;
size_t const enoughSpace = 12345;
params.new_file_size = enoughSpace;
boost::iostreams::mapped_file mmap{ params };

MmapAlloc alloc(mmap);
::flatbuffers::FlatBufferBuilder builder(params.new_file_size, &alloc);

// fill buffer

mmap.close();
@dbaileychess
Copy link
Collaborator

Can you create another memmap file afterwards at the correct location and size, and delete the original one?

@ridilculous
Copy link
Author

@dbaileychess can you elaborate on this? What do you mean with afterwards?

After closing mmap create a new mmap with the correct offset, e,g. hint = enoughSpace - builder.GetSize()?
Or leave the mmap open and create a 2nd one over it at the correct memory location?

The former wouldn't make sense i think and the latter did throw an exception within boost::iostreams::detail::mapped_file_impl::try_map_file.
Even if it would work, it would imply copying (most or) all data from one to another disk location, right?

@github-actions
Copy link

github-actions bot commented Jul 9, 2023

This issue is stale because it has been open 6 months with no activity. Please comment or label not-stale, or this will be closed in 14 days.

@github-actions github-actions bot added the stale label Jul 9, 2023
@github-actions
Copy link

This issue was automatically closed due to no activity for 6 months plus the 14 day notice period.

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Jul 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants