Sunday, January 09, 2022

Reading January 2022

Sunday, November 28, 2021

Reading December 2021

 



Monday, November 08, 2021

A Writeup on GraphQL and FHIR

GraphQL and FHIR 

I know, some of these links won't work. Internal hub and all...but assume they were informative.

During I+P I spent some of my time looking at a FHIR to GraphQL plugin from The Guild (Envelop) to generically map various known schemas, including  FHIR, to unopinionated GraphQL. e.g. a generic enough GraphQL schema that it can be done automatically instead of carefully traversing the source and  target and agreeing on a mapping. Downside of being unopinionated: your GraphQL schema can be a little hard to navigate because it's not in the client's  product/domain language and you're generally serving up your structure client side. Upside: with little effort you can stitch together / federate JSON,  REST, FHIR, mySQL, oDate, gRPC, and more. Downside...if you're generalizing your GraphQL instead of carefully targeting the client's language /expectations/ease of use and instead of carefully mapping for where you get duplicate calls, you can lose a lot of the power / advantage that comes with  GraphQL. 

I have some familiarity with FHIR (Fast Healthcare Interoperability Resources) and Rasanjalee Dissanayaka M. Dissanayaka Mudiyanselage does an  amazing, in-depth walkthrough here that puts my little bit of configuration programming to shame: Introduction to FHIR (Fast Healthcare Interoperability  Resource). But I started at square one by reviewing the internally recommended Udemy (free) course, Introduction to FHIR. One hour long and I'd like to  recommend you save yourself the hour and never watch it unless it's a requirement for your annual review I'll summarize it for you here: 

Introduction to FHIR (1 hr) – Udemy - https://www.udemy.com/course/introduction-to-fhir/learn/lecture/11088132#overview by Vivian Sendling-Ortiz 

Advice: I started by watching the opening parts on 1.25x (intro, JRE, Mirth install)…in retrospect, watch the whole thing on 1.25x or faster and be  prepared to skip ahead. The presentation is primarily config and transform and a lot of waiting on installs/minor changes. Of interest, what is FHIR: 

Fast Healthcare Interoperability Resources – an upgrade to HL7 

HL7 2.x was basically unreadable – pipe separated sections 

HL7 3.x was XML and a dismal failure 

FHIR supports XML and JSON 

All these versions and other specifications are out there and the mapping between them is generally already known. 

What was important? Mirth (NextGen Connect) exists. It is a cross platform middleware interface engine for exchanging messages. A glorified transformation engine that understands the transforms between standards. Very anticlimactic.…I felt like I was back in my  BizTalk days. However, Rasanjalee Dissanayaka M. Dissanayaka Mudiyanselage and I had a good discussion about Mirth because  she did some of her own transform work for a POC (see the post above). So as a talking point, something to know, it's useful and  moves you past simple - but potentially frequent - mapping to a dynamic process that gets a lot of table stakes transformation out of the  way and gives you a place to codify one-off rules. 

It's more fun to just play around with the installed Swagger 

http://hapi.fhir.org/baseR4/swagger-ui/ - the OpenAPi/Swagger for experimenting with POST/GET against the API. 

http://hapi.fhir.org/# - the base URL for HAPI FHIR. If you want to play the API, there's given and family and then there's name  that does a match against both (Scott for instance) and returns a nice list that you can then use to examine the JSON and start  to traverse the schema. 

Here are some pictures of all that in action...but not the Mirth/NextGen Connect transformer. I recommend going over to Rasanjalee's post (Introd uction to FHIR (Fast Healthcare Interoperability Resource)) as she walks through all of this including inserts, updates, persistence layer, data  structures, and more. What I call out below is a little- to no-code variation. 

Grabbing a patient via gender...note this isn't part of the add-in, exactly (it is), we'll get to that... 

A list that comes back from patient if I search on name = Scott (looking for matches in first || last)...

The response...

So WHAT does interrogating the FHIR online OpenAPI have to do with my dig into FHIR/GraphQL? Well, the add-in (you can grab the code here: https://gi thub.com/Urigo/graphql-mesh) is smart enough to allow you to simply point at that exact endpoint to consume the REST as a GraphQL structure. In a  perfect world, you'd have your own FHIR server, but for demonstration purposes, pointing at a working demo on the web is at least a few magnitudes  easier. Here's some of my YML pointed directly at that specification...in this case, the source is FHIR, but per above, there are add ins for other data  sources as well and if you install them and map them in this yml, you can federate your GraphQL calls. I'll leave the concerns about performance for you  to mull around in your head if you're doing multiple calls for a tree structure that reach to multiple locations for individual properties within a nested list.

We'll get to the good part in a moment, but another important bit are the scripts to grab the schema and create/map/generate the unopinionated GraphQL  schema from the latest FHIR version. Bonus....the addin serves up a local version of GraphiQL UI on localhost for you to play around with...don't worry  about that error message in the terminal It's not important unless you think it is and the FHIR server is having simultaneous issues and you assume  causation when it's just correlation. If you have the luxury, sometimes waiting is the best debugging.

Head over to your browser. Change Jane to Scott (in my case), click/unclick any properties or sub-objects you want to add which automatically inserts  them into the GraphQL JSON query (all conveniently saved as flat files with a graphql extension in example-queries), click the run button, and select the  option you want (PatientSearch in this case) and voila...back come Scott records and all their data via GraphQL:

I know what you're thinking...Scott, you didn't code bupkis. Nothing. You installed Node, cloned Mesh from Github, ran the install for the connector, and  ran the scripts that were laid out for you. Yep. And if you're lazier than I am you can even target the sandbox with files... https://codesandbox.io/s/github /Urigo/graphql-mesh/tree/master/examples/openapi-stackexchange?file=/package.json or the sandbox without files... https://9u5jz.sse.codesandbox.io/ 

So...what did I learn from my configuration adventure and why did I do it?  

Uri told me to in "Let’s Build! FHIR + GraphQL Mesh – Connect All Your Existing Sources into One Data Graph with FHIR and GraphQL Mesh"  (57:16) and I really wanted to see it for myself. It felt like he skipped some steps/parts because it was so easy and I wanted to know if that was  realistic or if it was more complicated than he was letting on: https://www.youtube.com/watch?v=bnLB98xQc9o - https://www.youtube.com/watch? v=bnLB98xQc9o 

The ability to federate multiple data sources with little to no effort (until you run into performance concerns) and map them all to a unified only what's-asked-for response is incredibly powerful. 

It moves the exploration of the FHIR specification client side and modifications to that specification (via OpenAPI, limited / sub-version / hosting  as a RDBMS structure) can be accommodated by regenerating the GraphQL schema (although keeping in mind rules around deprecating fields). I can interrogate the FHIR schema like it's JSON and see real results without many individual GETs. 

GraphQL provides numerous advantages over the REST implementation with complicated/nested schemas: 

Avoids overfetching (only the data asked for) 

Full intent in one query/one result 

Spec is typed 

Can “explore” client side using GraphiQL (or other tools) and you don’t have to be a developer to explore the data/queries. There are Server/Client versions…you can incrementally expand/build out the schema as you expand/build your data source or  handlers, and the schema isn't hidden in the code. 

GraphQL Mesh builds on GraphQL advantages: 

Uses existing services/standards/sources (SQL, GRPC, MQ, SOAP, OpenAPI/Swagger, oData, MSGraph, GraphQL….). And now, a  FHIR handler (SOAP handler + JSON schema handler + some extra bits). They're looking for a good option / use case for a MS SQL  handler. 

Uses the original GraphQL protocol but with a better dev experience (e.g. the whole "non-opinionated" aspect versus learning a new  data structure) 

Run/execute anywhere 

Federated: connections between different graphs (an example is an event, like a dev conference, with a city/location, and there’s a  separate city service with weather…all events can be returned in a response that includes weather as a property or subobject for each  event). 

Already handles mutations and will be getting more support. FHIR is a big focus because Uri himself was consulting with companies  going after easy FHIR implementations.


My Notes on GraphQL Conference 2021

 GraphQL Conference 2021 

Last week I attended (virtually) the GraphQL Conference 2021 (https://graphqlconf.org/). I didn’t get to attend the full event, but I thought I’d go over  sessions I did get to attend before other meetings cut me short: 

Keynote Panel: 

There are consistent GQL patterns emerging in caching, error handling, refetches, deprecated decoration/tags, and schema federation. Tools are  emerging to coordinate these standards (Apollo Studio, typescript integrations, plugins). Scott: when I worked GraphQL on an external analytics /BI project, we had to consider all of these without mature existing patterns in place. Each was a real challenge and a potential delay to product  delivery. 

It’s (still) gaining steam, Apollo just received $130 million in Series D funding they intend to spend on new devs. Scott: a friend of mine got a cold  offer just today and ask me if I had referred him (I hadn’t). 

Per Laurie Barth from Netflix, GraphQL moves a lot of decisions from the application to the client, but it also moves a lot of problems from the  application to the client. Scott: understandability, exploration, optimization of calls, much more. 

It’s a different mindset. Have to optimize for client-side development (per my note on that last bullet). 

Sean Grove – GraphQL as a Generalized Next-Gen API Client: 

Federation of Soap, OpenAPI, AsyncAPI, and GraphQL via an API Catalog: one graph queries. 

Proponent of spending time reading your API. 

Roy Derks – Headless 

Headless Schema Stitching: https://github.com/royderks/saleor-strapi-stitching 

https://github.com/gmac/schema-stitching-handbook 

Scott: At this point, it’s fairly obvious federation is paramount for many presenters/companies using GraphQL. 

Colby Fayock – Wordpress and GraphQL 

Less interesting for me personally, but a good walkthrough of the basics around how a GraphQL interface gets rid of a lot of the all-or-none  pattern that can cripple REST interfaces that are chatty (as was the case for WordPress pre-version 4.7). 

Tanmai Gopal 

Concept of batch and dedup in the GraphQL resolvers. 

More succinctly, you get better gains federating the data instead of the services. Avoid overlap, repeat…optimize. Per below, your data  structure may need to be tailored in some ways to your GraphQL schema and the resolvers/logic may need some unusual optimization as well to  avoid waste. 

Min Chen and Karthik Balasubramanian – Automated Schema Generation from ER-Based Service Modeling 

Linkedin optimizations to use declarative ER-based service modeling to create the schema. 

Combination of GET/FINDER and extensions to get performant arrays (basically on tables before joins happen) where GraphQL might otherwise  trigger a deep lookup before resolving.

My last job our GraphQL schema was auto-generated by the code with some decorators. We didn’t get as far as flattening the ids to increase  performance, but we didn’t lockstep schema changes against code initially – code moved and schema followed automatically. Did mean we had  to move code around and regenerate when we considered (and updated) the “usuability” of the schema structure for external clients. Per the speakers and per my own experience, data source structure is crucial and may have to complement your GraphQL implementation for  performance and usability. 

They did have to address gaps in the GraphQL definition language and their implementation. 

Uri Golshtein – GQL to the Next Level: 

I was excited about this one. I have a FHIR-GQL presentation of his queued up to watch this week. 

Works for Envelop – create plugins for GraphQL (https://the-guild.dev/blog/introducing-envelop

Of the opinion most schemas are too opinionated and the more you can generalize and let tools handle it, the better. Scott: it’s interesting,  because sometimes “opinionated” makes it much easier for the client to understand. There’s a balance. If the client doesn’t use it because it’s  non-intuitive, then it has little value even if it’s less opinionated. 

Solve the hard problems once, use anywhere. 

Frameworks for frameworks.


Burning Bridges

Based on the lack of a response, I think I've burnt my bridges at Facebook.  Honestly, it's a bit sad because I like to think both my management and GraphQL skills are pretty good and they create some amazing tech - but I didn't mince words after the second exchange:

<name>, I would love to work for you as a company that has such cool tech. But I have to say....as someone who understands you have an audience of three billion and can't get your %^%% together when it comes to being a responsible entity in the world....I'm concerned.  I really don't think I can be part of that....I can barely stand to talk to friends on FB knowing the implications for the world politically. simply because we're sharing memes and humor and that supports your visibility for antisocial upvoting.  Let me know when your ML is cleaned up and doesn't cause global stress. I fully understand I could help fix your problems ethically, but I haven't seen a commitment to resolving the problem/s.


As someone who works in ML, I get it's a challenge and there are hard problems, but it's a challenge a company has to step into, not talk away.  I truly hope FB finds an ethical sweet spot

Secret Santa

There is a LOT of garbage in this code. Notes to myself, print statements to debug, etc.  But this is the current incarnation of the secret santa code I'm using.  The old XML/Outlook version in C sharp was better, but I ran into email issues with Office moving online and switching employers and ending up on a Mac. Now I just generate .txt files.

2022 updates should include getting rid of the separate checks for exclusions and telling it to check for anyone in the full exclusion list so I can make it ragged.

Here....target == who[1] or target == who[2] or target == who[0]

And there's a lot of unnecessary iterative garbage to make sure it can brute force it's way to a list when it gets toward the end where it might dead end itself.  But to be honest, that doesn't take 100 iterations, it's over in like four. Anyway...I think I can boil this down to a dozen lines of code, so maybe I'll do that next year. Personally, I'm just glad I was able to find it again after moving machines twice in 2.5 years.

#0 fix the code >>

#0a put the guts of the retry in a function that can error out with an error message

#0b and let the error message run a global retry

#  load >> run >> each person >> one person fails >> clear and run

#1 import from a settings file (don't need a UI although the last one had one)

#2 which means we need to write to a settings file

#   name, spouse, last year (use the format year(last)file.txt as the default

#   use JSON instead of XML this time), ah....need an email address!

#3 put it in a bucket (file) in S3, put the code in a python lambda

#4 autoemail - lambda send email with ses: https://aws.amazon.com/premiumsupport/knowledge-center/lambda-send-email-ses/

 

 

#2020 list...this needs to generate from input/output

#e.g. that last value should be a read from last year's file

listWho = [("Scott","Jen","Jackie")]

listWho = listWho + [("Eryn","Scott","Jen")]

listWho = listWho + [("Oliver","Jackie","Andrew")]

listWho = listWho + [("Jen","Scott","Ceri")]

listWho = listWho + [("Andrew","Jackie","Jen")]

listWho = listWho + [("Jackie","Andrew","John")]

listWho = listWho + [("Allison","Ceri","Andrew")]

listWho = listWho + [("Ceri","Allison","Ellen")]

listWho = listWho + [("John","Ellen","Scott")]

listWho = listWho + [("Ellen","John","Allison")]

 

 

 

def LoopLoop(listWho):

    targetExclusion = []

   

    for who in listWho:

        #print (len(listWho))

        #print(who)

        print(" ")

        print(" ")

        print("Owner: " + who[0])

        myRand = random.randint(0,len(listWho)-1)

        #print("random: " + str(myRand))

        target = listWho[myRand][1]

        #print("target: " + target)

        myIter = 0

        #could put all this in a function and call for success or failure

        #eg while false...

        #target can't be used already

 

        while target == who[1] or target == who[2] or target == who[0] or target in targetExclusion:

            target = listWho[random.randint(0,len(listWho)-1)][0] #I think it was my [1] here that was the mess, never returned somne folks

            print("temp target: " + target)

           

            myIter = myIter + 1

            #print(myIter)

            if myIter >=100:

                print("I FAILED")

                return False;

                #break #kill it and we will start over

 

        targetExclusion = targetExclusion + [target]

        print(targetExclusion)

        print("target: " + target)

        f= open(str(datetime.now().year) + "_" + who[0] + ".txt","w+")

        f.write("YOUR SECRET SANTA RECIPIENT IS: " + target)

        f.close()

        #do it again

        #actually, turn this into the loop

    return True; #ah, this is where I"m failing, I have to get to len to return true

 

 

print(listWho)

mybool = False

print(mybool)

while mybool == False:

    print(mybool)

    mybool = LoopLoop(listWho)

 

print ("Mybool = ")

print (mybool)

print ("end of script")

Reading November 2021

Yeah, yeah.  Blah blah. I didn't update the last reading list.  I will...I've got lots of stuff read, I'm just slow to update.  11/28/2021, I'm closing this month down early.  I have some other things I'm part way into, but they can just make my December list.