C based cross platform lockless data structures

Filed in General Leave a comment

Came across this today  , I’m particularly interested in the lock-free ring-buffer / and/or queue as it will be useful in some work I’m doing on a processing pipeline. Makes me think I should consider publishing some of the data structures I have created…. I’ll need to tidy them up somewhat but I’ll think about it, they might be useful to someone :)

will go through this lib properly later on – much to do recently….

Linking libcURL statically in an application

Filed in C++ | Code | cURL | visual-studio-2005 Leave a comment

Had to jump through hoops today to get this to link with our product, the DLL version is much easier and I had done all testing up to this point via that route. However it looks like the client is going with a web service based product and I had to link statically because we don’t want to have to distribute cURL libs along with our product.

So first off – linking with the DLL is pretty easy because you basically include the libcurl_imp.lib  (or libcurld_imp.lib ) and then bob is your proverbial

So I changed the lib to be the static version in both release and debug configurations and that’s when the fun started. I got issues with missing symbols during the link:

E.G.

There were more but they were all to do with the “easy” suite of functions.

This is caused because I needed to define the fact that my application was linking statically by defining the following flag on the compile command line :

or in properties->Configuration Properties->C/C++->PreProcessor:

props_pre

This is symetric, and probably changes something in the headers I would guess changing the way that the API is implemented in my application as well as how it is compiled into the library or DLL.

That then got me issues with my runtime libraries when I attempted to rebuild. This can be sorted by making sure that the runtime used by your application is the same as the runtime in the libCurl project:

runtimelib

Once that was done I still had issues with missing symbols E.G.

Again there were others, but they were all to do with LDAP. This was sorted by rebuilding the libcURL library again with another define:

That was it for me, although if you do actually need LDAP support then you need to add

to the link line of your application.

I did read that you could also get issues with socket libs. This is not an issue for me because I am performing socket operations myself and therefore already link with

However if you get undefined symbols of a winsock type nature then you’ll probably need to link with this as well.

 

 

 

Beginning Android apps and Emulators.

Filed in Android | Code | General | Samsung Galaxy Note 10.1 Leave a comment

I’ve spent the last few days gradually working through this  to get the basics of creating Android applications. I have downloaded the latest Android SDK including a ready made eclipse environment.

About Dialog from the installed eclipse.

About Dialog from the installed eclipse.

The installation comes with various perspectives, views and tools for handling the ADK and the development process. These tools hide a lot of set up that is required under the covers.

ADK_menu

The menu (above) contains buttons to get to the SDK management tool, the Virtual Device Manager, a wizard for adding the XML files required by an application, and a lint tool for identifying issues with your code and config files.

 

ADKLink

ADK_AVDM

APITool

 

After setting up an Android Virtual Device (AVD), you get an emulator to run your application on

AVD1 AVD2

The emulator was extremely slow at first but after some reading I found that I needed to add in an extra package that supports the GPU and virtual machine acceleration :

Accellerator

There is also a flag in the AVD edit screen to switch on the GPU accelleration. This is misleading because it is still there even if the package above is not installed, and the resulting  AVD doesn’t start…

AVDEdit

 

I’ll write more as I go on, it appears that the coding for the apps themselves is fairly easy, combinations of XML and code however can be problematic – leading to things either not working, or not working as expected. The book and the IDE do not match up either, but this is making me think harder about what it is saying and therefore helping in a strange way so far…

New toy! Samsung Galaxy Note 10.1

Filed in Android | General | Java | Samsung Galaxy Note 10.1 Leave a comment

Probably the best Thing about This new tablet (at this particular point in time!)  is the handwriting Recognition and just how accurate it is….

2012-12-25 17.43.52

But there is a problem with uploading data…. l needed to access the above Screenshot from drop box and add it into my library. It may be that I simply didn’t try to upload from my “local” drive now I think about it… Will investigate my options on that because if I can upload directly from my tablet that would be excellent…

I played with some of the other features as well since initially posting this, and I think If I were a student again I would want one of these (over an iPad).  I can imagine taking notes and being able to create some excellent reference material for revision etc. As it stands I can also see myself using this to create designs and document things at work as well – the pen is a great addition to the product and it is a feature that makes this tab standout from the iPad – even if there are a lot of similarities.

Another thing here is the ability to develop software for this without the need for a yearly subscription to Apple to do so, and the need to use Objective-C (although it is very nice). However I will need to refresh my acquaintance with Java as this appears to be the recommended route over C++.

The Android SDK is here

Boost Property Trees, cURL, Web services, Google Map API distances

Filed in Boost | C++ | Code | cURL Leave a comment

I’ve recently used boost::property_tree::ptree    at work  and realized they actually have quite a lot of utility beyond what the name infers. The documentation is here . The 3 problems recently that  needed to be solved were due to the integration of third party services into an existing core component.

  1. Google Maps API v.3 support was required
  2. Support for a Web Service based Mapping API  from Vendor X
  3. Conversion of XML based data exported from one system converted to proprietary format.

After much deliberation on the best method to interface with both Google and the Web-Service provided by the vendor, I discovered libcURL. It is a very powerful library (written in C) which implements the plumbing required to do common network based requests supporting many protocols.

libcurl is a free and easy-to-use client-side URL transfer library, supporting DICT, FILE, FTP, FTPS, Gopher, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMTP, SMTPS, Telnet and TFTP. libcurl supports SSL certificates, HTTP POST, HTTP PUT, FTP uploading, HTTP form based upload, proxies, cookies, user+password authentication (Basic, Digest, NTLM, Negotiate, Kerberos), file transfer resume, http proxy tunneling and more!

libcURL is easy to use in a C++ program although it does have limitations with respect to concurrency. But initially at least the usage is light and if we need to do things faster I will revisit this.

The second thing I used was boost::property_tree::ptree  I found it via several SO questions where the class was being promoted for reading XML. It turned out that it has several different ways of serialization / de-serialization including INI files, XML and JSON. Its original use was as a generic way to handle config files as the name suggests, but the mechanism was so general it is easy to use it for other purposes.

It is implemented using RapidXML under the hood which is a header only XML parser available independently of Boost here. It is meant to be fast and so is not fully featured, but it is very functional for the majority of basic tasks you might want to do.

As I said above (3) was the conversion of data in XML format to another format, and I was interested in very little of the original document. So after reading the file I processed the property tree as below:

And then wrote out the data I wanted. It was a 10 line program in the end. Sure a script could have done as much, but I realized I could reuse this in the actual code for the product and it would be simple and fast.

I am currently getting a SOAP response back from the web service from Vendor X and I basically put that directly into a property tree also.

The beauty of this method is that it is almost exactly the same code for Google as Vendor X and so I had both products running within our framework within hours of each other. It would have been quicker had I not had some Licensing issues with Vendor X that manifested themselves as strange (but valid) results.

Google required that I craft some URLs to extract the data that was required (incomplete code)

 

I will post some details of how I used libcURL in another post as this is meant to be fairly general and more focused on the property trees and their use.

 

 

rvalues and std::move : questions

Filed in C++ | Code Leave a comment

What do we get for free?

I ended up having a few issues with my program in the last post, firstly to do with getting the basic function working. I expected the compiler to generate the move constructor for me and in fact If I had been more lax and not defined the copy constructor myself it would have actually done so.

from here 

Yes, they are, in §12.8.9:

If the definition of a class X does not explicitly declare a move constructor, one will be implicitly declared as defaulted if and only if

  • X does not have a user-declared copy constructor,
  • X does not have a user-declared copy assignment operator,
  • X does not have a user-declared move assignment operator,
  • X does not have a user-declared destructor, and
  • the move constructor would not be implicitly defined as deleted.

and §12.8.10

The implicitly-declared move constructor for class X will have the form

X::X(X&&)

Similarly for the move assigment operator. However because of the nature of the program and my expectations to see less messages generated from the copy constructor I realised the issue and worked through what the problem was.

What happens to those objects?

The second thing that is now apparent comes from the last output of the program. In this output it appears that the initial object I created using the const char*  constructor (assigned id 0) is destroyed many times through the program run. It is of course the temporaries generated along the way that are being destroyed, and the meaningful object is actually being passed from scope to scope. But the issue then becomes what happens to those temporaries?

In my previous example the data is simple and no real side effects will be seen because I don’t explicitly do anything in the destructor. However what if there were some resource management in that class and the destructor invokes some clean up? If the objects are moved through a stack of calls, at each level there will be destructors invoked which would invoke those operations.

This line of questioning has highlighted the fact that I made a mistake in the implementation which I will now correct:

The problem lies with the move constructor above. Whilst I set up the new object from the old, I never explicitly clear the old object. Thats why we see object(0) being destroyed multiple times.

What we need to do here is actually deactivate the old object so that if it were handling resources of some kind, we effectively render it inert. Stack Overflow had a question that addresses the best way to implement the move constructor which I believe gives the best idiomatic way to approach this:

Notice the other issue with my earlier code, this explicitly calls std::move()  for each member.

The real change though is the setting of the members in the donor object so that its destruction is safe.

 Revisiting assignment – right or wrong?

My assignment operator  implementation was driven by my use of the copy and swap idiom to ensure exception safety. The problem(?) is that we want to do a move to avoid extra allocations, and swap will effectively do an extra construction to effect the swap

At least that is what I expected. It does a move construction , then move assigns the two objects into the correct position (vc11).  So my original move assignment operator

is probably good.  There is a lot of discussion on SO about this particular area but this answer provides an interesting take and the whole page is interesting in general. I need to think more about this because I am still not sure if what I did is an error or not.

 

 

 

Minimising copies during function calls – rvalues and std::move()

Filed in C++ | Code Leave a comment

N.B. there are mistakes in the code below which I discuss in the next post. The reason they exist is that this site is more like a journal and so is fairly raw in content since its intended audience is me.

I have been trying to understand how the new features of C++11 are going to affect the code that I write.  I have read anecdotal evidence that the new move/rvalue functionality would help in writing more natural code.

 This Article describes the new rvalue references in C++11.

The following is a program that I wrote to understand how the objects are copied through the function calls. It is a simplified example of how some production code I am working will behave. The following is a non POD class  that simulates the data object that will exist in the real system.

The code that follows uses the class Bar . Basically at the heart there is a  queue<Bar> which contains instances of the  class Bar . This is populated indirectly via 2 layers of function calls. I began using a naive approach of passing by value because it is the simplest implementation.

The above is called by the following main function

The output with the pass by value approach is below.

In all the original object is copied 5 times through the process. The first optimisation I would do is to pass by reference through to the push into the queue. This will remove copies (1) & (2) above but there will be a copy made when the object is pushed. So now we are moving objects into the queue fairly efficiently. The return route still makes 2 copies before the return.

Bar t = w.WFooFunc2();  in the main function should on the face of it should make another copy, however it doesn’t because of Return Value Optimisation (RVO) or elision. This process means that the compiler will omit a copy of the object when returning from a function by value. It may not be possible if the function has multiple return points, or can return one of several objects.

So at this point I am looking at introducing the new move features of c++11.

The Bar object is modified to include a move constructor – much like a copy constructor, and move assignment operator. These are implemented in a similar way to the copy and assignment operator but are more efficient as they do not construct the object fully, they move the contents of the source into the new object leaving the old object as a shell – easily destroyed.

Now we can see that simply adding these operators (actually just the move constructor) has made a further saving over the last version without any change in the code at all. The Temporary created in the return before is now a move operation.

The second copy above is made when the local is copied from the return value of  front() . The value is copied because a reference is returned and so the compiler cannot know what the expected lifetime of that object is. It cannot see that the next operation (the pop() ) will cause it to be destoyed.

We can make a further change which brings into play a new function from the standard lib which forces the comiler to do the move:

This then leaves us with :

Now we only make the copy when the object is pushed into the queue. So can I get rid of that as well? The answer is yes by adding a further call to  std::move() when I push the object passed by reference

It turns out that I have made a slight mistake in this code that could have a bad effect in production code. I had instinctively realised but couldn’t place what the issue was. I started to talk about this in the next article and thinking about the situation made it clear what the issue is.

 

Minecraft Mesh Toolbox

Filed in JavaScript | OpenGL Leave a comment

Minecraft Mesh Toolbox. Java script based OpenGL tests on Voxel rendering – I get 60fps in Chrome…

Hello world!

Filed in C++ | Code | General | OpenGL Leave a comment

Replaced the original version of the site because of strange incompatibilities with some of the plugins I wanted. Hopefully this will now work properly.

The syntax highlighting plugin I’ve gone with finally is from here  the main reason being the fact that I don’t have to change code myself (E.G the includes have angle brackets) and it integrates into the editor very nicely.

, ,

TOP