null responses not being handled

Topics: Bug / Problems
May 17, 2016 at 8:55 AM
Hi.

When using the JSON source component, I have a problem with null responses. I'm not sure if the component should handle null responses, or if I should be handling the error that is generated by the component from within SSIS.

I'm querying a service which is not publicly available (not very helpful I know, sorry) to check whether a user exists. If the user exists, I ignore it - and if it doesn't exist I want to carry on and do some other action (involving creating the user). At the moment I can check whether the user exists (the subject of my other post) and handle the response correctly if the user does actually exist. But if the user doesn't exist I get a null response, which the JSON Source Component doesn't handle. I get the following 3 errors:
[Check if user exists] Error: Error reading JObject from JsonReader. Current JsonReader item is not an object: Null. Path '', line 1, position 4. at Newtonsoft.Json.Linq.JObject.Load(JsonReader reader, JsonLoadSettings settings)
at Newtonsoft.Json.Linq.JObject.Load(JsonReader reader)
at com.webkingsoft.JSONSource_Common.JSONSourceComponent.ProcessInMemory(StreamReader sr, RootType rootType, PipelineBuffer inputbuffer, PipelineBuffer outputbuffer)

[Check if user exists [8]] Error: System.Exception: Error occurred: Error reading JObject from JsonReader. Current JsonReader item is not an object: Null. Path '', line 1, position 4. at Newtonsoft.Json.Linq.JObject.Load(JsonReader reader, JsonLoadSettings settings)
at Newtonsoft.Json.Linq.JObject.Load(JsonReader reader)
at com.webkingsoft.JSONSource_Common.JSONSourceComponent.ProcessInMemory(StreamReader sr, RootType rootType, PipelineBuffer inputbuffer, PipelineBuffer outputbuffer)
at com.webkingsoft.JSONSource_Common.JSONSourceComponent.ProcessInMemory(StreamReader sr, RootType rootType, PipelineBuffer inputbuffer, PipelineBuffer outputbuffer)
at com.webkingsoft.JSONSource_Common.JSONSourceComponent.PrimeOutput(Int32 outputs, Int32[] outputIDs, PipelineBuffer[] buffers)
at Microsoft.SqlServer.Dts.Pipeline.ManagedComponentHost.HostPrimeOutput(IDTSManagedComponentWrapper100 wrapper, Int32 outputs, Int32[] outputIDs, IDTSBuffer100[] buffers, IntPtr ppBufferWirePacket)

[SSIS.Pipeline] Error: SSIS Error Code DTS_E_PRIMEOUTPUTFAILED. The PrimeOutput method on Check if user exists returned error code 0x80131500. The component returned a failure code when the pipeline engine called PrimeOutput(). The meaning of the failure code is defined by the component, but the error is fatal and the pipeline stopped executing. There may be error messages posted before this with more information about the failure.
So the question is 3-fold: Should the component be handling null responses? Am I using the component incorrectly? Should I be handling the error within SSIS somehow (if so, any pointers?)?

Thanks for looking at this.

Cheers
Wally
Coordinator
May 17, 2016 at 9:39 AM
Hi,

I see your point. At the moment the component will not handle this kind of unexpected null objects. In the future revisions I am planning to add another output lane for errors, but honestly at the moment I'm pretty busy and this won't be done before Setptember 2016.

A possible solution would be to give 0 rows as output when errors occur, but this might confuse other people: letting the app crash when an error happens is usually a better way to go. For this reason I would not do that, but if you wanted, you might recompile the component adding a check for the StreamReader in JsonSourceComponent.cs at line 384, avoiding to call ProcessInMemory if the StreamReader is null. Then the solution for your problem is just a check on the number of rows outputted by the component.
May 18, 2016 at 8:33 AM
Hi

Your suggested solution would be ideal in this case. I can understand why you might not want to make it the default behaviour, but in my case it would be perfect.

I downloaded the source and while changing the code to match that behaviour (especially after you pointed out exactly what I needed to do and where to do it) was not difficult, I have a whole pile of errors in the project. A lot refer to missing assembly references - some of which I can easily resolve - others I don't know what I'm doing...). The other two main problems I had were "The name 'ComponentMetaData' does not exist in the current context" (or other component), and "no such method to override" - which interestingly only happened with VS2015, and not in VS2013 - leading me to think maybe I'm just using the wrong Visual Studio version?

Anyway, this is obviously beyond my capabililties, and I'm not expecting a tutorial!! I can write a console app to do what I'm wanting to do and at this stage it's probably the best approach, but it'd sit better in SSIS. I'm relatively new to SSIS and to be honest, I'm surprised it doesn't support JSON out of the box. Oh well.

Thanks
Wally
Coordinator
May 18, 2016 at 8:47 AM
WallyAnchor wrote:
Hi

Your suggested solution would be ideal in this case. I can understand why you might not want to make it the default behaviour, but in my case it would be perfect.

I downloaded the source and while changing the code to match that behaviour (especially after you pointed out exactly what I needed to do and where to do it) was not difficult, I have a whole pile of errors in the project. A lot refer to missing assembly references - some of which I can easily resolve - others I don't know what I'm doing...). The other two main problems I had were "The name 'ComponentMetaData' does not exist in the current context" (or other component), and "no such method to override" - which interestingly only happened with VS2015, and not in VS2013 - leading me to think maybe I'm just using the wrong Visual Studio version?

Anyway, this is obviously beyond my capabililties, and I'm not expecting a tutorial!! I can write a console app to do what I'm wanting to do and at this stage it's probably the best approach, but it'd sit better in SSIS. I'm relatively new to SSIS and to be honest, I'm surprised it doesn't support JSON out of the box. Oh well.

Thanks
Wally
Hi Wally,

I might be able to add a "checkbox" allowing to bypass errors, such as "Do not crash on errors", that would be unchecked by default. That shouldn't be too hard to implement, but would require couple of weeks (at the time I am really busy so I am only dedicating couple of hours per day to fix existing bugs).
If you do not have any urgent need, you can wait for that.

Cheers,
Alberto.
Coordinator
May 18, 2016 at 8:48 AM
This discussion has been copied to a work item. Click here to go to the work item and continue the discussion.
Coordinator
Nov 6, 2016 at 11:43 PM
Hi!
latest version should fix this problem. When the object is not found, NULL is outputted.
Marked as answer by webking on 11/6/2016 at 4:43 PM