Sanjay Bhagia's Blog

We all should be bit more like dogs

Fri, 29 Mar 2024 00:00:00 GMT

Dogs. They are adorable. They are cute and cuddly and bring so much joy to our lives.

Take my dog, Chihu - oh, I adore her. I love spending time with her.

I’m not writing this to convince you to get a dog for yourself. What I’m proposing, however, is that we should all be more like dogs! Yes, I know, it’s unusual but hear me out.

Calmness Dogs are calm (for the most part). They bring a sense of stillness in you when they are around. There is a certain peace you feel in their presence. Right now, when I'm writing this, she sits beside me, calm and composed. Such a positive energy
Satisfaction with very little They are happy with very little. You give them their food, spoil them with a few treats and spend some time with them, and they couldn’t be happier! Who would have thought a tiny piece of carrot could be so satisfying?

It doesn’t take much for them to feel happy and content; it shouldn’t take much for us, either.
No overthinking They don’t complicate their lives with overthinking. It's like they have cracked the code that overthinking doesn't lead to anything good. Why do we tend to overcomplicate everything in our lives?
Masters of solitude They are good at handling boredom and staying with themselves. Something we have lost to the modern world! When was the last time you sat with yourself and did nothing? Give it a try; you might find a new person within you.
Quick to forgive They don’t hold grudges. They do something stupid; you get mad at them, they get bit scared. But they will quietly sneak into your lap after few minutes. Matter resolved! Almost a zen-like quality that we can learn from them.
Unmatched excitement They are the most excited to see you when you are coming home. Every single time, without a fail! This sense of excitement on little things and appreciation of what you have got, something we can learn from these voiceless creatures.
Make you move They help you move. Yes, you have to take your dog out for some physical work, good for them and good for you. Maybe, maybe you can reach your daily steps goal in the process.
What you see is what you get I haven't seen dogs pretending to be someone they are not. They are what they are. They emote how they feel. You can see through them. I understand we can't always be like that, but what if we try a little bit more to be ourselves and more authentic?

So, wouldn't incorporating or amplifying these dog-like qualities enhance your daily life? It has certainly helped me. You be the judge of it.

Do you have a dog? What do you like about them? Share your stories in the comments below.

Remember, sometimes the greatest lessons come from the most unexpected teachers.

Until next time.

Handling Order Processing with Azure API Management and Service Bus Topics

Tue, 10 Oct 2023 00:00:00 GMT

Scenario

Imagine we are are a dropshipping supplier for e-commerce stores around the world. Clients receive orders from their end-customers and forward them to our backend to process and deliver them. We are going to automate the process of receiving orders by exposing APIs. In order to distribute the load, we are providing the endpoints to receive orders based on the geography. For example, we have an endpoint to receive orders from Sydney, Australia etc.

Here is the high level architecture of the system we are going to design in this blog post.

We will use Azure API Management to expose the endpoints and make use of Azure APIM policies to send the orders to Azure Service Bus Topics. Service Bus Topic will have different subscriptions to receive orders for different geographies. At the end, we will have multiple consumers (in the form of Azure functions) to process the messages from the Service Bus Topic for respective subscriptions. This architecture, allows us to scale the processing of orders based on geography. We can have multiple consumers to process the messages from the Service Bus Topic to distribute the load.

Before we dive into the details, let's understand the components we are going to use in this demo.

Service Bus Topic

Azure Service Bus is a fully-managed enterprise integration message broker. It provides reliable message delivery between applications and services, even when one or more of the endpoints are offline. Service Bus supports a variety of messaging patterns, including publish/subscribe, request/response, and message queuing. It also provides features such as message batching, dead-lettering, and message deferral. Service Bus can be used to decouple applications and services, enabling them to scale independently and evolve over time.

One of the core components of Azure Service Bus is the Topic, which can be used for publish/subscribe scenarios. Messages are sent to a topic and delivered to one or more associated subscriptions.

API Management

Azure API Management (APIM) is a fully managed service that enables customers to publish, secure, transform, maintain, and monitor APIs. It provides a unified experience across the entire API lifecycle, enabling developers to create APIs and app developers to consume them easily.

API Management Policies

Azure API Management policies are a set of rules that are executed by the API Gateway when a request is made to an API. These policies can be used to modify the request or response, enforce security, or perform other actions. Policies can be defined at the API level, the operation level, or the product level. Azure API Management provides a rich set of built-in policies and also allows you to define custom policies using XML and C# (policy expressions).

When integrating APIM with Azure Service Bus Topic, the idea is to allow APIM to publish messages to the Service Bus Topic by using one of the built-in policies. This makes is very easy for us to send messages to the Service Bus without having to write any code.

Provision the Infrastructure

Let's walk through the steps to provision the required infrastructure. I'm using Azure CLI (az cli) to provision the resources. You can use the Azure Portal, ARM templates, or PowerShell to do the same.

Set the account

# Login & set appropriate subscription
az login
az account show
az account set --subscription <subscription-id>

Create a resource group

We will create all of the resources in a single resource group.

# Define a variable which we will use later
resourceGroupName=azecommstore

# Create the resource group
az group create --name $resourceGroupName --location australiaeast

Create Azure API Management Service

Here, we will provision the consumption tier of Azure APIM, create the API, and set up a operation (endpoint) that will receive requests from clients.

For this demo, I'm leveraging system-assigned managed identity to establish secure communication between APIM and Service Bus.

The --enable-managed-identity flag is going to create a System Assigned Managed identity for the APIM. You could create the user-assigned managed identity and use that as well. In fact, I would recommend that. However, there is no support for creating user-assigned managed identity using Azure CLI, you would have to use ARM templates, PowerShell, or the Portal.

# Define variable, which we will use later
apimName=azecommstoreapimae

# Create APIM instance with managed identity enabled
az apim create --name $apimName --resource-group $resourceGroupName --location australiaeast --publisher-name Sanjay --publisher-email admin@gmail.com --sku-name Consumption --enable-managed-identity true

Create API

Now that we have the APIM instance ready, let's create an API that will receive orders for us.

az apim api create --service-name $apiName -g $resourceGroupName --api-id Orders --path '/orders' --display-name 'Orders'

Create API Operation

Once we have the API, let's create a POST operation that will receive the request from the client and send the message to the Service Bus Topic.

I'm creating a generic endpoint here (e.g., /process/{geography}), so that we can have as many URLs for geographies (e.g., /process/sydney, /process/melbourne etc.) without defining separate operations.

az apim api operation create --resource-group $resourceGroupName --service-name $apimName --api-id Orders --url-template "/process/{geography}" --method "POST" --display-name 'Process Orders' --description 'Process Orders for E-Store'

Create Service Bus Resources

Here, we will create the Azure Service Bus namespace, topic, and subscription. We will also create a rule on the subscription to filter messages based on the originUrl header (will explain a bit later about this). This will allow us to route the messages to different subscriptions based on the originUrl header.

Namespace

# Define variable, which we will use later
serviceBusNamespace=azsbnsorders

# Create a namespace with standard SKU
az servicebus namespace create --name $serviceBusNamespace --resource-group $resourceGroupName --location australiaeast --sku Standard

Service Bus Topic

az servicebus topic create --name orders --resource-group $resourceGroupName --enable-partitioning false --namespace $serviceBusNamespace

Service Bus Topic Subscription

We can create multiple subscriptions to forward messages as they come to our topic. I'm creating a subscription for processing orders coming for Sydney.

az servicebus topic subscription create --name sydneyorders --resource-group $resourceGroupName --namespace-name $serviceBusNamespace --topic-name orders

Service Bus Topic Subscription Rule

The way we route messages to subscriptions is via filters. Here, we are creating a SqlFilter based on the originUrl custom property that will be coming with incoming messages to this topic.

az servicebus topic subscription rule create --resource-group $resourceGroupName --namespace-name $serviceBusNamespace --topic-name orders --subscription-name sydneyorders --name processorders --filter-sql-expression "originUrl='/orders/process/sydney'"

Assign Permissions to Service Bus Namespace

In order for APIM to send messages to the Service Bus Topic, we need to assign the Azure Service Bus Data Sender role for APIM instance (system-assigned managed identity we created earlier at the time of provisioning the APIM instance) on service bus namespace.

# Get the subscription id
subscriptionId=$(az account show --query id --output tsv)

# Get the objectId of the APIM instance (system-assigned managed identity)
assigneeId=$(az apim show --name $apimName --resource-group $resourceGroupName --query 'identity.principalId' --out tsv)

# Finally, assign the permissions on Azure Service Bus namespace
az role assignment create --role "Azure Service Bus Data Sender" --assignee $assigneeId --scope /subscriptions/$subscriptionId/resourceGroups/$resourceGroupName/providers/Microsoft.ServiceBus/namespaces/$serviceBusNamespace

For the sake of simplicity, I'm setting the permissions on the entire namespace. However, you can be very specific (e.g., set the permissions on the topic or subscription level) by changing the scope of the role assignment.

--scope /subscriptions/$subscriptionId/resourceGroups/$resourceGroupName/providers/Microsoft.ServiceBus/namespaces/$serviceBusNamespace/topics/$service_bus_topic/subscriptions/$service_bus_subscription

Set Policies in APIM

This is where the magic happens. We are leveraging Azure APIM policies to receive requests, extract information from incoming requests, and send messages to the Service Bus Topic. We are using the managed identity to get the access token from Azure AD and then using that access token to send the message to the Service Bus Topic.

For the purposes of demonstration, this endpoint is not secured. We haven't set up any authentication or authorization. In a real-world scenario, you would want to secure this endpoint. You can configure the setting on the API level or you can validate the incoming request within the policy itself.

Here, we are simply setting the backend URL to the service bus namespace.

Navigate to APIM -> APIs -> Orders -> Process Orders -> Inbound processing -> Policy Code editor.

Copy and paste the following policy and click Save

<policies>
     <inbound>
        <base />
        <!--Azure Service Bus-->
        <!-- Set the backend URL to Service Bus Namespace-->
        <rewrite-uri template="orders/messages" />
        <set-backend-service base-url="https://azsbnsorders.servicebus.windows.net" />
        <!-- Custom Headers are sent as 'Custom Properties' on service bus messages -->
        <set-header name="originUrl" exists-action="override">
            <value>@(context.Request.Headers.GetValueOrDefault("X-Original-URL", ""))</value>
        </set-header>
        <authentication-managed-identity resource="https://servicebus.azure.net" output-token-variable-name="msi-access-token" ignore-error="false" />
        <set-header name="Authorization" exists-action="override">
            <value>@((string)context.Variables["msi-access-token"])</value>
        </set-header>
    </inbound>
    <backend>
        <base />
    </backend>
    <outbound>
        <base />
    </outbound>
    <on-error>
        <base />
    </on-error>
</policies>

Test the API

Now, we have everything in place. Let's test the API. We will use Postman to test the API.

Let's issue a simple request from Postman to the endpoint

https://azecommstoreapimae.azure-api.net/orders/process/sydney

As we can see, the request is successful and we received a 201 Created response!

Curl Command

For reference, here is the CURL request

curl --location 'https://azecommstoreapimae.azure-api.net/orders/process/sydney' \
--header 'Content-Type: application/json' \
--data '{
    "key": "value"
}'

Let's validate the message from the Portal.

Azure Service Bus - Portal

As we can see, we did indeed receive the message in our subscription within the Service Bus Topic. Upon further inspection, we can see that the message was sent to the sydneyorders subscription and the processorders rule was applied.

From this point, you can very well create an Azure function to receive messages from this topic and subscription.

Bonus - Azure Function App

Now, we are ready to write our code to process the orders. But first, let's spin up an azure function. We need a storage account and Azure Function app first.

Storage Account

az storage account create --name azecommstorestorage --resource-group $resourceGroupName --location australiaeast --sku Standard_LRS

Azure Function

az functionapp create --resource-group $resourceGroupName --consumption-plan-location australiaeast --name sydneyordersprocessor --storage-account azecommstorestorage --runtime dotnet-isolated --functions-version 4 --os-type Linux

Write and Test Code

You can use Azure Function Core Tools or your favorite IDE to create a function app. Here is the simple function I have created. It simply logs the message it receives.

using Microsoft.Azure.Functions.Worker;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;

public class ServiceBusTopicListeners
{
    private ILogger _logger;
    public ServiceBusTopicListeners(IServiceProvider serviceProvider)
    {
        var loggerFactory = serviceProvider.GetRequiredService<ILoggerFactory>();
        _logger = loggerFactory.CreateLogger(this.GetType());
    }
    [Function(nameof(ProcessOrders))]
    public async Task ProcessOrders([ServiceBusTrigger("orders",
        subscriptionName: "sydneyorders", 
        Connection = "servicebus_cs", IsSessionsEnabled = false)] 
        string message, FunctionContext context)
    {
        _logger.LogInformation($"Received message: {message}");   
    }    
}

When we run this function locally in debug mode, and set the breakpoint, we see that message is received successfully.

Clean up the resources

Always a good idea to clean up the resources you are not using

az group delete --name $resourceGroupName

Source Code

You can find the complete source code of this post on my GitHub repo. Add the connection string of your Azure Service Bus namespace in local.settings.json to test this out locally. I have also listed all az cli command in this blog post in one file infra.sh for your reference.

Conclusion

In this blog post, we saw how we can leverage Azure API Management policies to send messages to Azure Service Bus Topic. We also saw how we can leverage Managed Identity to get the access token and use that to send messages to the Service Bus Topic. This approach makes it easy to send messages to the Service Bus Topic without having to write any code.

Hope this was helpful.

Cheers

Hands-on with Katonic.ai's Generative AI Platform

Wed, 27 Sep 2023 00:00:00 GMT

OpenAI has taken the world by storm, and Generative AI is all the rage these days. It's not just OpenAI; many other companies are working on Generative AI and Large Language Models (LLMs). These models are rapidly changing how we interact with machines and build applications. From Copilots to Spotify's recent Voice Translation for podcasters, we are witnessing the sheer power of these models penetrating our daily lives.

In this blog post, I'm exploring one platform I have recently encountered that offers a no-code environment to create, deploy and manage such Large Language models. Let's dive in!"

What is Generative AI?

In case you are not familiar with Generative AI, I asked our trusted friend ChatGPT and this what it had to say:

Generative AI is like a super smart robot that can make up new things, like stories, pictures, or even music. It learns from lots of other things people have made, and then it can create its own stuff that's brand new and sometimes really cool! It's like a creative friend who can make up fun games and drawings for you to enjoy.

What are Large Language Models (LLMs)

Large language models (LLMs) are a type of artificial intelligence model designed to understand and generate human-like text. They are built using deep learning, specifically a kind of neural network called a transformer.

Large language models are like super-advanced text generators that can produce coherent and contextually relevant text based on the massive amount of data they've been trained on. Katonic.ai

Katonic.ai

Now that we are on the same page let's see what Katonica.ai is and what it brings to the table.

Before discussing Katonic's Generative AI Platform, let's touch base on Katonica to set the context for the rest of the discussion.

What is Katonic.ai?

Katonic is a collection of cloud-native tools for all the stages of the Model Development Life Cycle (data exploration, feature preparation, model training/tuning, model serving, model testing, and model versioning). Katonic provides a unified UI and has tooling that allows these traditionally separate tools to work seamlessly together. An important part of this tooling is the pipeline system, which enables users to build integrated end-to-end pipelines that connect all components of their Model Development Life Cycle (MDLC).

Katonic is for data scientists and engineers looking to build production-grade machine learning implementations. Katonic can run locally in your development environment or a production cluster. Often, pipelines will be developed locally and migrated once the pipelines are ready. Katonic provides a unified system—leveraging Kubernetes for containerisation and scalability for the portability and reusability of its pipelines (source)

Katonic Generative AI Platform

The Katonic Generative AI platform allows users to instantly deploy and manage their own Generative AI models. It offers categories like Chatbot, Classify, Extract, Summarize, and Generate, along with a Tuning Studio for creating customised prompts. The platform also provides a seamless deployment process, turning models into production-ready APIs with just a few clicks. Lastly, it offers a Playground where users can explore and interact with various Generative AI models.

Katonic offers a no-code Generative AI Platform that is simple and intuitive, allowing for rapidly building powerful Enterprise-grade AI Applications without coding.

Why would I need to run my own LLMs if I can simply use something like ChatGPT?

It's a good question, so let me share a few thoughts about this.

Data Security & Privacy If you have spent time in this space, you would be aware that this is one of the most talked about concerns. Large Language Models, by nature, are trained on large data sets. When we talk about enterprises or businesses where data is susceptible, we cannot hand it over to the model running in the public domain. Running a model in a controlled environment ensures our data doesn't leave our premises. Even if it leaves, we can anonymise or sanitise it such that no private information goes out.
Cost Depending on your usage, running your model after the initial investment might be more cost-effective, as you won't be paying per API call.
Latency Running a local instance can result in lower latency since you don't need to make an API call over the internet. This is especially beneficial for real-time applications.
Custom Training You can fine-tune the model specifically for your domain or dataset, making it specialized for specific tasks or industries, which can improve its accuracy and relevance for specialized applications.
Others
- Pinning specific versions of the model
- Bypassing API rate limits while using third-party APIs
- Tight control of the deployment and uptime of the system
- Better control over integration with your line of business applications.

This is not an exhaustive list, but you get the point. There is a place and good reason for running LLMs in your environment that you have complete control of.

Take it for a spin.

Enough talk, let's take this for a spin and see how can we use this platform.

You will need a Katonic.ai account. You can create a free account here. Once you log in, you land on this Generative AI page.

UX is neat and intuitive; you won't need much guidance, but I'll walk you through here anyway.

Create a project

For demonstration purposes, I'll create a chatbot application trained on some of the blog posts I have written in the past. Once the chatbot is ready, we will ask a few questions to see how it performs.

Alright, so hit the + Create a new Generative AI Project button.

Selecting project type

Fill-in the project name and description. The important thing is to pick the type of project we want to create. This makes it easy to choose because we should know our use case without getting too much into technicalities. I'm picking up Chat here.

Add Data

The next screen is where we have to provide the source of data we want our chatbot to be trained on. We can either provide URLs or upload documents. I'll provide five links to the blog posts that I have written on Pulumi. (5 is the limit for free project).

Customise

This is where things get interesting. Here we can tweak a number of settings. First up is the language model we want to pick. In the free tier, we can only select Pick the Best LLM. Katonica picks up the appropriate model based on the project type we selected earlier. But we could choose any other LLM model.

Next up, we can customise the look and feel of our Chat application we will get in the end. And also if we want to make this application be available as public or private. This is pretty powerful, so you can be sure your enterprise applications are not accessible publicly.

It is not enough to train the models on just the data; we want to control it so that it doesn't blurb out the answers in any way it pleases. We want to ensure our models don't include inappropriate information, especially if this is something we are using in a business context. For example, we want to filter our content of any abusive wording. Katonica allows us to tweak these things at this stage, and it'll ensure the generated responses respect the constraints we set up here. Other than content filtration and restrictions, we can also define the personality our bot should have while answering the questions. This is your opportunity to get creative; if you have always wanted to sound like William Shakespeare, go ahead and tell the model!

Project Completion

Once we have provided our data and customised the model to our wishes, hit next. Click the Deploy button and let Katonica work its magic. It might take a few minutes for the model to be ready. Once it's ready, you'll see the green tick, and now you are ready to play with it. You can head over to 'Deploy' tab from the left to see all your models.

Hit the Try this 'Chatbot' button to launch your very own bot!

Testing

Here, we have our very own chatbot trained on some of my blog posts in all its glory! Notice, here I did bit of customisation (though I was a bit lazy and just changed the colour to orange; but you can change the logo, description, colour, etc., to make it your own).

Alright, it's time to test it. Let me ask a few questions

Not bad! It pulled relevant information from my blog posts. Let's try few more questions.

As you can see, it picks up the content from the links I provided and gives me correct answers. It even returns the formatted results (bullet points) while answering. Since it's an LLM, you can converse with it as you please and it will return the results based on your input.

Well, this was pretty sweet and quick. In a matter of minutes, I have my very own Large Language Model trained on my data with my defined constraints and personality, running as a web application - all without writing a single line of code! 🤯

Access the model via APIs

This is impressive, but this might not be something you are after. You may want to use the APIs to integrate the model into your application. That is a fair point, and Katonica's Generative AI platform provides access via APIs! Let's see how we can use it.

Get API URL and Access Token

If you head over the Deploy screen or 'Generative AI' tab, you will find your deployed chatbot model.

Click on API, and it'll open up a modal. Here, we can see two endpoints:

Model Prediction API endpoint
Model Feedback API endpoint

Let's create the API token before we click the first link, as we will need it for the next step. Click Create API token, provide the token name and click the Create token button.

Copy this token, as you won't be able to see this again.

Now, select the API usage docs tab and click the View documentation button

It should bring up the swagger page.

Click Authorize and paste the API token we copied earlier.

Use the API

Set the following payload as request body. This is the same question we asked earlier in the chatbot UI that was created for us.

{
"data": "what are the supported secrets
  management providers? List them as bullet points"
}

Press the Execute button and look at the response body. The same response we got earlier.

This is how easy it is to use the APIs for the project we created. With this, we are not restricted to using the UI that was created for us, but we can use the APIs within our existing applications to integrate the power of LLMs.

You can also check the usage and monitor your application from the application card.

Playground

So far, we have covered how we can create a simple chatbot using the no-code approach of Katonica's Generative AI Platform. We trained the model based on our sample data by providing a few URLs, and we could ask questions. We also used the APIs to achieve the same result. This shows that you can use the APIs and integrate them within your application to talk to the model rather than using the UI provided by Katonica.

Let's switch gears and explore some other interesting features that the platform has to offer.

If you recall, while creating our chatbot early on, we let Katonica choose the best model for us.

And it's fine. Katonic picks up the model suitable for the type of application we want to build. It makes it very easy for us if we don't understand much about different models or how they function.

However, if we want to gain more control over the type of LLM model we want to work with, Katonic makes it extremely easy for us to do so.

Playground is a place where we can explore a variety of Generative AI models (60+ at the time of this writing available on the platform).

Here, we can find models such as Stable Diffusion, code Llama, MusicGen and many others. You can navigate this list and filter based on categories for easier navigation.

Let's explore stable-diffusion model here.

As we can see, based on our input, an astronaut riding a horse on mars, hd, dramatic lightning, it generates the image on the right.

Besides the apparent input, we can change and control other parameters, such as image dimensions and other model-specific settings. Change the settings and see the difference in the output.

Another important element on this page is the 'Model Card'. For each model, you can find the details of what it does, how it works, and how you can use it, along with examples and other important information on the model. This is very handy as you can find all the necessary information at one place while you are testing any model.

Tuning Studio

Another stellar feature, albeit an advanced one, is the Tuning Studio.

Here, you can easily manage all your custom prompts for various models you want to use for different applications. Katonica offers pre-created prompts that you can use as a foundation before customising them for your use cases.

This is a good place to experiment with your prompts and tweak the parameters of the desired model (even try out different models to see which gives you the desired outcome). You can provide instructions to the model, give it some training data and test out the models.

Once you are finished with your prompts and happy with the results, you can save it!

Tuning Studio also lets you manage your models so that you can manage them in one place.

Takeaway

I'm impressed by the simplicity of the platform and yet how powerful it is to create, manage and operate. It makes it very convenient to develop models from a various options and deploy them in public or private environments. This becomes a one-stop shop for testing different models, both proprietary and open source. I can see how enterprises can easily train these models based on their data and deploy and manage them without compromising security. We are going to see more and more capabilities coming in this space in the future but Katonica's no-code, private and secure platform definitely seems a great fit for companies to quickly leverage the power of LLMs within their line of offerings.

You can read more about it on their official website.

Thanks for sticking to the end. I hope you find this helpful.
Cheers

Deploying IronPdf to Azure Functions via CI/CD using Azure DevOps

Sun, 20 Aug 2023 00:00:00 GMT

You can find the source code for the sample I am using in this blog post on my GitHub. This contains the code for Azure Function along-with YML pipeline that I'm going to discuss here.

Brief Introduction to IronPdf

IronPDF is the leading C# PDF library for generating & editing PDFs. Its user friendly API allows developers to rapidly deliver professional, high quality PDFs from HTML in .NET projects. Some of the key feature are:

Convert web forms, local HTML pages, and other web pages to PDF with .NET
Allow users to download documents, send them by email, or store them in the cloud.
Produce invoices, quotes, reports, contracts, and other documents.
Work with ASP.NET, ASP.NET Core, web forms, MVC, Web APIs on .NET Framework, and .NET Core.

You can read more about it on their official website

Setting the context

In this blogpost, I'm going to focus on deploying your code that uses IronPdf to an Azure Function. I'll be using Azure Function on Windows, but you can find the guidance on Linux on their website as well.

For the purpose of demonstration, I've created a simple HTTP Triggered Azure Function that receives HTML string in request and returns the PDF File. I assume, the function app is already created.
Here is how the generated PDF looks like.

Deploying to Azure Function via Azure DevOps

Now we have a functioning code, let's deploy this to Azure via CI/CD pipeline using Azure DevOps

The CI/CD pipeline is similar to any azure function pipeline, there is no major difference except choosing the right deployment method.

Choosing the Deployment method.

In Azure DevOps, for the AzureFunctionApp@1 task, there are the following supported methods.

auto
zipDeploy
runFromPackage

auto is the default method, when you add a task. But for IronPdf, zipDeploy is the recommended mode

YML Snippet

Here is the yml snippet for this step:

steps:
- task: AzureFunctionApp@1
  displayName: Azure functions app deploy
  inputs:
    azureSubscription: <subscription name>
    appType: functionApp
    appName: ironpdf-azurefunc-app 
    package: $(Pipeline.Workspace)/azurefunction/a.zip
    deploymentMethod: zipDeploy
    appSettings: >
      -FUNCTIONS_EXTENSION_VERSION "~4"

The only different is setting the deploymentMethod to zipDeploy instead of default auto.

This will deploy the code from your solution in Azure Function.

You can find the complete CI/CD pipeline in my GitHub repository here. You will need to adjust the settings to make it work for your environment such as subscription name etc.

Bonus: GitHub Action

If you prefer using GitHub, I have also included the action for your in my repository. You can find it here

Hope it helps. Cheers

The best investment I made in 2022

Mon, 06 Mar 2023 00:00:00 GMT

We all know that 2022 was a challenging year for many of us around the world. The stock markets struggled, inflation reached record highs, and the fear of a global recession loomed large. I was no exception and didn't fare well with financial investments last year. However, despite all the challenges, there is one thing that I did in 2022 that turned out to be the best investment for me. And it only cost me $15!

It was a bullet journal.

You may have heard about bullet journal before, but if you haven't, it's a simple journaling approach that can help you organize your life. You can read all you want about the bullet journal here. I started with a very basic approach, using it to plan my weekly tasks. Rather than planning each day, I found that planning each week worked better for me. I primarily use this approach to organize my personal life, not my work (for work, I use a similar weekly planning approach but on OneNote).

At first, it may seem like a simple concept, but I have achieved much more by just planning my weeks over the last year than I ever have before. Here are some of the things I accomplished in 2022:

More workouts. I have been much more consistent than I have ever been in my entire life.
More running. In fact, I completed my very first 10k run run back in September 2022.
More walking. I completed my very first 28k walk walk in October 2022, in addition to walking my dog almost every day.
More socialization with friends.
Hosted more events at my home.
Published my very first course on cloud computing fundamentals, yeah!
More experiences, including trying out new restaurants, attending events, concerts, exhibitions, and outdoor activities.
More done around the house, including fixing things and maintenance tasks.
More certifications. I completed several certifications in 2021 and had to renew most of them in 2022, so with a little bit of planning, I managed to get them done.
Increased my financial literacy by taking few courses to educate myself on the subject.

All of this was achieved simply by putting down the tasks I intended to do each week in my bullet journal. No overthinking, no fancy productivity hacks, no over-the-top system to get things done. Just a simple list of activities that I planned to do each week.

One of the best things about bullet journaling is that you can use any tool that works for you, whether it's an analog journal or a digital equivalent. What matters is the act of doing it.

So, if you're not already using a bullet journal or a similar system, I highly encourage you to give it a try. You may be surprised at how much you can get done by being intentional about your tasks. Here's to the next year and all the things we'll achieve! 🥂

Pulumi - State Management

Mon, 01 Feb 2021 00:00:00 GMT

I am learning Pulumi and I'll be sharing my journey along the way here in the form of blog posts and code samples on my GitHub. Join me on this journey if this topic interests you.

Disclaimer: I have hands-on experience working with the Microsoft Azure Cloud platform not so much with others. So in here, I'll be looking at Pulumi from Azure's lens and most of my references and examples will be with Azure.

If you haven't read my previous blog posts around this topic, you can check them out here:

In this blog post, I'm going to explore and understand how Pulumi manages the state.

What is State?

Pulumi manages the state (in the form of a state file - a JSON file). This is the source of truth for Pulumi. Whenever you make any change in your program, Pulumi refers to that state and checks with resource providers to determine what action needs to be taken (update, delete, add or nothing). Each stack in Pulumi has it's own state file.

Pulumi keeps this state file in a backend. By default, pulumi offers it's own free backend service (http://pulumi.com), it works seamlessly and offers a nice UI on top of it where you can see what's going on during the lifecycle of any program and stack. However, as mentioned earlier, we can take complete control of the state management. This is where, we can either keep the state locally or in any other backend such as Azure Storage (which we will explorer in here a bit later). You can read more about the advantages of using Pulumi's hosted backend here

Using Azure Storage as a backend

You can use any backend (e.g., Local file system, AWS S3, Google Cloud Storage etc.) to manage your state but for this blogpost let's focus on Azure Blob Storage.

For this, we need to first create a storage account and a container within that. You can perform all of the following operations via Azure Portal if you feel comfortable with that.

Create a resource group

az group create --name rg-pulumi-state-dev --location australiasoutheast

Create Storage Account

Create an account

$SA=$(az storage account create --name pulumiazstate --resource-group rg-pulumi-state-dev --location australiasoutheast --sku Standard_LRS --access-tier Hot --https-only true --kind StorageV2)

Extract keys

$KEYS=$(az storage account keys list --account-name pulumiazstate --resource-group rg-pulumi-state-dev --output json)

Set Environment Variables

We need to set AZURE_STORAGE_ACCOUNT and AZURE_STORAGE_KEY as environment variables so that the CLI can use them while performing other operations. You can set them manually or via script. Here is the PowerShell / CLI combination to set them.

[System.Environment]::SetEnvironmentVariable('AZURE_STORAGE_ACCOUNT','pulumiazstate') [System.Environment]::SetEnvironmentVariable('AZURE_STORAGE_KEY',$(echo $KEYS | jq -r .[0].value)

Create a container in the storage account

az storage container create --name state

the command above uses the 'AZURE_STORAGE_KEY' environment variable we defined earlier.

Head over to the Azure Portal to see the account and a container has been successfully created.

Set storage account as state backend

Now we have a storage account and a container ready, let's tell Pulumi to use that as a backend. Pulumi offers following command to do this:
pulumi login azblob://<container-path>

so go ahead and execute pulumi login azblob://state

Create a new stack

Create a new directory mkdir azbackend and navigate cd azbackend

From this point onward, the rest of the approach is similar to what we have done earlier. The only different being when you create a stack, you will be asked for a passphrase.

pulumi new

You can use the Azure key vault to manage the encryption if you will. Refer to my previous blogpost to see how that can be done.

Now we have a stack created with our backend managed within Azure Blob Storage. If you head over to the container we created earlier, you will see a folder for the stack we created 'dev' and a JSON file.

This is the checkpoint file or state file Pulumi uses where it keeps track of the state of our infrastructure. You can see the file contents directly from within the portal in edit mode.

Preview the stack

If you go one step further and execute pulumi preview you will find everything works as usual. Notice, the URL is now pointing to our blob instead of Pulumi's backend.

Now, go ahead and execute pulumi up and let it finish. If you check the state file now, you will see a lot more content in there.

Logout

In order to work with external storage we used pulumi login azblob://<container-path> earlier. If you would like to switch to a different backend or return to the default one, you can use pulumi logout

This simply logs you out and you can start with pulumi login again with the choice of your backend.

Interacting with the state via CLI

You can use some commands to interact with the state, such as: pulumi state pulumi state delete pulumi state unprotect

If you would like to export the sate you can use pulumi stack export and pulumi stack import allows you to import the existing state to your stack.

Things to be aware of

Pulumi's default hosted backend service offers quite a several things that make our life easier. So when we decide to user any external backend systems, we loose some nice features that are worth being aware of.

No UI

First and foremost, if you use any external backend, you can't use the nice UI that pulumi.com offers. Which means, you will have to read the JSON file if you want to make sense of what is going on.

Security

Since state file contains encryption keys etc, it is our responsibility to secure the access to the storage account. Be sure you follow the principle of least privilege when it comes to setting up backends.

Stack names

when we use Pulumi's backend service it creates stack names as fully qualified names (projectname/stackname). Pulumi handles that behind the scenes and we get a good experience of having a separation of the project name for stacks. When we are using any other backend, we don't get that. Stack name is simply the name we mention at the time of pulumi new. So if we created a stack with dev already, we won't be able to create another stack with the same name. This forces us to set the strategy of naming our stacks from the get-go. To put it simply, Pulumi used fully qualified names for naming stacks (organization/project/stack) where as any other backend uses just the stack name. It is up to us to introduce the hierarchy.

Summary

Pulumi provides a robust and free hosted service to manage the state and backend but at the same time we can complete control over that if we want. We can use a file system, AWS S3, Azure Blob Storage and Google Cloud Storage to manage the backend by ourselves. The tools pulumi provides (CLI etc.) work similarly irrespective of the backend you use. However, you need to be aware of few things you might have to manage on your own if you choose not to use Pulumi's hosted backend service.

Cheers

Pulumi - Secrets Management

Tue, 26 Jan 2021 00:00:00 GMT

I am learning Pulumi and I'll be sharing my journey along the way here in the form of blog posts and code samples on my GitHub. Join me on this journey if this topic interests you.

Disclaimer: I have hands-on experience working with Microsoft Azure Cloud platform not so much with others. So in here, I'll be looking at Pulumi from Azure's lens and most of my references and examples will be with Azure.

If you haven't read my previous blog posts around this topic, you can check them out here:

In this blog post, I'm going to explore and understand how Pulumi manages secrets.

We will build on the Configuration Management blog post that I wrote earlier so please have a look if you haven't done so already.

Having sensitive information such as Passwords, API Keys, Tokens, ConnectionStrings etc. is quite common for any application. However keeping them in your source code in plaintext is never a good idea, let alone checking it in your source control.

Pulumi's configuration system lets us add secrets to our configuration in a similar way we manage any other configuration. Pulumi offers multiple approaches to manage sensitive information and exposes a number of APIs and tools we can use to work with them seamlessly, so without further a do, let's explore how it all works.

Handling Secrets

First off, let's see how can we set secrets in our configuration.

We use pulumi config set with --secret flag to set a secret

pulumi config set --secret mySecretKey mySecretValue

Check the config pulumi config

To retrieve the secrets programmatically, the use of RequireSecrets variant is recommended.

Pulumi.Log.Info($"mySecretKey: {config.RequireSecret("mySecretKey")}");

see the value is masked

If we look at the Pulumi.dev.yaml file in VSCode, this is what we see:

config:
  azure:location: australiaeast
  hello-pulumi:data:
    active: true
    nums:
    - 1
    - 2
    - 3
  hello-pulumi:location: australiasoutheast
  hello-pulumi:mySecretKey:
    secure: AAABAFybCK/w34qectPQHPWSoivUhG7uI5//y7Xqc/6Cczh3k3Yi35zvrk+U
  hello-pulumi:resourcesName: pulumi

Looking at this, what we see is that the configuration we set as a secret is not stored in plaintext, it is in fact encrypted. By default, Pulumi uses per-stack encryption key managed by the Pulumi Service to encrypt values.

However, this might not be desirable in situations where you want to manage the state by yourself by using any external storage like S3 bucket, Azure Blob storage. Or you simply don't want Pulumi to manage the keys and would prefer keeping that control and management with you. For such scenarios, Pulumi supports multiple providers for secrets management.

As of this writing, these are the supported providers:

default
passphrase
awskms
azurekeyvault
gcpkms
hashivault

We used default provider in our examples above. passphrase allows you to keep the encryption keys to yourself but still use Pulumi's backend service for state management OR you are self-managing the state storage (e.g., Azure Storage). The last option is where you are completely managing your own keys (bring your own key a.k.a BYOK)

Using Passphrase

Let's create a new stack in our codebase that we used in the previous blogpost (grab the core here)

pulumi stack init passphrasestack --secrets-provider passphrase

or create a new project pulumi new azure-csharp --secrets-provider passphrase

It'll ask for the passphrase. This is the passphrase that Pulumi is going to use to encrypt the configuration system.

Looking at the project in VSCode, we can see a new file has been created

Let's add two keys (one normal and one secret) pulumi config set normalKey normalValue pulumi config set --secret secretKey secretValue

This is how our stack file looks like

If you pay attention, the secure value in Pulumi.passphrasestack.yaml is different to what we had in Pulumi.dev.yaml

Notice, we need to enter the passphrase every time we are performing any operations. This can be avoided by setting PULUMI_CONFIG_PASSPHRASE in your environment variables (can be done on the local machine or in the CI system).

Using Azure Key Vault for encryption

This would be a similar approach for any other cloud-based provider but we are looking at Azure Key Vault here. In order to use the Azure Key Vault for managing the encryption keys, the approach is same i.e., we set --secret-provider to Azure Key Vault via the CLI

$ pulumi stack init <name> --secrets-provider="<provider>://<provider-settings>"

e.g., $ pulumi stack init <name> --secrets-provider="azurekeyvault://<keyvaultname>.vault.azure.net/keys/<key name>"

Before we could do this, we need to create the Key Vault and a Key in the vault that we need.

Create Azure Key Vault

az keyvault create --name pulumiazvault --resource-group rg-pulumi-dev --location australiasoutheast

Create the Key in Vault

az keyvault key create --name pulumiencryptionkey --vault-name pulumiazvault

Assign Access

When you create the vault and key from the CLI, a policy is created by default but it doesn't set 'encrypt' and 'decrypt' policy for us, so use this command to include that.

Extract Object Id of your account (you can pull this from Azure AD or run the command below to extract it. Use your account here)
az ad user show --id <your account name>

Look up for ObjectId of your account

az keyvault set-policy --name pulumiazvault --object-id <Object Id from above> --key-permissions encrypt decrypt get create delete list update import backup restore recover

I prefer using Azure AD Application instead of any individual user account for such scenarios. In that case you will need the ObjectId of the application instead

Now we have the key vault with a key that we will use for encryption along with the right access policy set.

Set Environment Variables

You can set this if you want to use the credentials from the CLI (that you already have set up). AZURE_KEYVAULT_AUTH_VIA_CLI=true

Checkout Azure Environment Authentication if you wish to use other authentication mechanism such as Client Credentials, Username/Password etc.

I prefer using Client Credentials since I'm using Azure AD Application approach

Create a new stack

I'm creating a new project for this scenario rather than using the previous one but you could very well create a new stack in the same project that uses Azure key Vault provider.

pulumi new azure-csharp --secrets-provider="azurekeyvault://pulumiazvault.vault.azure.net/keys/pulumiencryptionkey"

Pulumi.dev.yaml files looks like this after creating the new project. As you notice, we have the information about secrets provider and encryption key. From this point onward, Pulumi uses this information to encrypt the configuration system for us.

secretsprovider: azurekeyvault://pulumiazvault.vault.azure.net/keys/pulumiencryptionkey
encryptedkey: N1JsSnFjWGpKUl9jY3U1SlRQd2J2WXd3TEw0XzQxeWExY2hTOEhtLTRKWkd3cTViblg3OWF4VHFTcDY4amo4RDc4RUxyT3poNC0tUEVSM0ppbUxfcUdiN1dOWm9Cb0VrNU5TODF6dUlMVEg3c2VVc1ItTVZfRjFBQWpmdGNiX29xYU1BdTN2d0NlMmtWUWFUWUc0NjNJMnh4eUNybzdJMzlXZHdXMVRpODh4TnZfdDM1R2pfdS1rS3VlQlh5cDI4OXZmMkxXaFFuNzNBckJaRU9vcnl5WmJPeUVXMHpJSzlIOEZXOG5WTnl6akFXYmxWYkkyYVB6c3BsUWpDOGw5OFRJNXFyV0JBSjAybHVkTE9UMGF6MllqSW4xbm9GVW1pODRsbVJZbEh6OThPLUdBemlhcWp4V1dKaE5vN0NDeEFZdFM2cXk3emozSDI0UHlGQllESTdR
config:
  azure:location: australiasoutheast

Rest of the things works the same as default or passphrase provider for encryption.

Switching Encryption providers

You change the secrets provider of any of your stack anytime via this command. pulumi stack change-secrets-provider "<secrets-provider>" It's pretty straightforward, once the provider is changed, Pulumi will use the new secrets provider to encrypt configuration and state files.

Summary

Pulumi provides a robust mechanism for configuration management of your infrastructure. It provides support for handling secrets as well. You have the possibility to use external service providers for secrets management such as Azure Key Vault by using a familiar approach. The default option is secure and configurations and secrets are encrypted by the keys that Pulumi manages for us however we have the flexibility to take ownership of this and go to that extra mile for enhanced security or compliance reasons.

You can find the source code for this blog post here

Cheers

Pulumi - Configuration Management

Fri, 15 Jan 2021 00:00:00 GMT

I am learning Pulumi and I'll be sharing my journey along the way here in the form of blog posts and code samples on my GitHub. Join me on this journey if this topic interests you.

Disclaimer: I have hands-on experience working with Microsoft Azure Cloud platform not so much with others. So in here, I'll be looking at Pulumi from Azure's lens and most of my references and examples will be with Azure.

If you haven't read my previous blog posts around this topic, you can check them out here:

In this blog post, I'm going to explore and understand how configuration management works in Pulumi.

Understanding the Basics

In Pulumi each stack can be configured separately. By default, when using pulumi new a stack is created at that time and the file is named as Pulumi.[stack name].yaml. However, you can create new stacks with pulumi stack init command at any time. It's important to notice that with pulumi stack init no new file is created, this is something we need to create manually.

Stack configuration files are used to store stack variables and secrets (more on this in another blog post). We can interact with configuration management either via CLI or programmatically. Let's check them out.

Working with Configuration via CLI

Pulumi CLI provides pulumi config to work with the configuration systems.

pulumi config set <key> [value]: sets a configuration entry <key> to [value]

pulumi config get <key>: gets an existing configuration value with the key <key>

pulumi config: gets all configurations for the current stack. you can specify -s <stack name> flag to retrieve configurations for a particular stack.

Pulumi uses the project name as the default namespace and appends it as a prefix for key names. So pulumi config set location australiasoutheast will end up adding an entry (key/value) to Pulumi.dev.yaml file like this (where hello-pulumi is the project name):

config:
  azure:location: australiaeast
  hello-pulumi:location: australiasoutheast

This (namespace) makes it easier to segregate the configurations for different providers or components.

So if we have to retrieve the key from the specific namespace, we can specify the full name by adding prefix while fetching e.g., pulumi config get 'azure:location' or pulumi config get 'hello-pulumi:location'

Working with Configuration via Code

Working with configuration via code is also very straightforward. For C#, Pulumi exposes Pulumi.Config type.

var config = new Pulumi.Config();
var resourcesName = config.Get("resourcesName");
var location = config.Require("location");

Pulumi.Config() takes the namespace name as an optional parameter. If you don't pass it uses the project name

Using Config.Get will return null if the configuration value was is provided whereas Config.Require raises an exception with a helpful error message to prevent the deployment from continuing until the variable has been set.

Pulumi also exposes Pulumi.Deployment.Instance that allows us to retrieve the project name and stack name etc. at runtime Pulumi.Deployment.Instance.StackName and Pulumi.Deployment.Instance.ProjectName

Structured Configuration

Often we have structured configuration such as a JSON or YAML that we need to deal with in our applications. Pulumi works wonderfully well with that type of configuration as well.

We use the same pulumi config command to interact with the configuration system. The only difference with structured configuration is we need to pass --path switch such as pulumi config set --path 'data.active' true

So let's set some configuration using the CLI.

pulumi config set --path 'data.active' true
pulumi config set --path 'data.nums[0]' 1
pulumi config set --path 'data.nums[1]' 2 
pulumi config set --path 'data.nums[2]' 3

This results in following configuration.

config:
  azure:location: australiaeast
  hello-pulumi:location: australiasoutheast
  hello-pulumi:data:
    active: true
    nums:
    - 1
    - 2
    - 3

To retrieve the setting via the CLI, we use pulumi config get --path 'data.nums[2] and it returns 3

For programmatic approach, Pulumi exposes Config.GetObject and Config.RequireObject for such functionality

var config = new Pulumi.Config();
var data = config.RequireObject<JsonElement>("data");
Pulumi.Log.Info($"Active: {data.GetProperty("active")}");
Pulumi.Log.Info($"Nums[2] number?: {data.GetProperty("nums")[2]}");

Get vs Require works in a similar fashion we discussed previously.

Making changes to our program

In the previous blog post, we created a hello world pulumi program that created a resource group and a storage account for us (check out the source here). That time the resources were assigned unique names by Pulumi (this is the default behavior if we don't specify Name property of any resource). Let's change that and assign names explicitly to the resource group and storage account.

The naming convention I want to use is:
Resource Group: 'rg-pulumi-dev' (where 'dev' is the environment that we pick from the stack name).
Storage Account: 'strpulumidev'

In order to pick up the stack name at runtime, we can use Deployment.Instance

var stack = Deployment.Instance.StackName;

Now, let's define a new property called resourcesName by using the following command in our terminal:
pulumi config set resourcesName pulumi
This adds a new key/value pair in our dev stack. Once we have it, let's read that in our code.

var resourcesName = config.Require("resourcesName");

Lets use this information to construct the resource name at runtime

var resourceGroupName = $"rg-{resourcesName}-{stack}";

Now, we expand our resource group resource declaration to this:

var resourceGroup = new ResourceGroup("resourceGroup", new ResourceGroupArgs()
        {
            Name = resourceGroupName, //If we don't set this, Pulumi generates a random unique Id by itself
            Location = location
        });

Repeat it for storage account.

// Construct storage account name
var storageAccountName = $"str{resourcesName}{stack}";
var storageAccount = new Account("storage", new AccountArgs
{
    ResourceGroupName = resourceGroup.Name,
    Name = storageAccountName,  //If we don't set this, Pulumi generates a random unique Id by itself
    AccountReplicationType = "LRS",
    AccountTier = "Standard",
    Tags =  {
        { "Environment", "Dev" }
    }
});

Now when we do pulumi preview we can see what Pulumi is going to create for us by checking state over at pulumi.com

Summary

Pulumi provides a robust mechanism for configuration management of your infrastructure. You can interact with configs via the CLI and programming model with the same ease. There are a number of built-in utility functions within the SDK to make it easier to work with configurations. You can use the power of programming language to work with the configurations like you normally would in your application.

You can check out the entire source code along with configuration file here for reference.

Cheers

Pulumi - How it works?

Mon, 21 Sep 2020 00:00:00 GMT

I am learning Pulumi and I'll be sharing my journey along the way here in the form of blog posts and code samples on my GitHub. Join me on this journey if this topic interests you.

Disclaimer: I have hands-on experience working with Microsoft Azure Cloud platform not so much with others. So in here, I'll be looking at Pulumi from Azure's lens and most of my references and examples will be with Azure.

If you haven't read my previous blog posts around this topic, you can check them out here:

Pulumi - What and Why?

In this blog post, I'm going to explore and understand How Pulumi works. What are the underlying core concepts and toward the end, will add a 'Hello World' to see in practice.

How Pulumi works?

So let's try to understand how Pulumi actually works. Let's have a look at the Pulumi Engine Diagram that is take from their website.

Pulumi Engine Diagram from Pulumi.com

You write a Pulumi Program which is executed by a language host
You handover that Program to the Engine (using the Pulumi CLI)
The engine compares the desired state of your infrastructure with the stack's current state and figures out the needed changes
The engine then uses resource providers to manage the resources

Let's try to understand these concepts a bit more.

Core Concepts

Pulumi Programs

Pulumi programs are basically any program you written in any supported language of your choice (JavaScript, TypeScript, C#, Python, Go ... ). Pulumi provides SDKs for supported languages and you install the required packages (nuget, npm etc. ) to work with your providers besides the libraries you use for native languages

Desired State

State is at the core of all this. Pulumi manages the state (in the form of a state file). This is the source of truth for Pulumi. Whenever you make any change in your program, Pulumi refers to that state and check with the resource provider to determine what action needs to be taken (update, delete, add or nothing).

Backends

Pulumi provide two approaches to manage the state

The Pulumi Service backend (a free hosted service that Pulumi offers. https://pulumi.com )
A self-managed backend, either stored locally on your filesystem or remotely using a cloud storage service By default, Pulumi uses it's hosted service backend. This makes it almost trivial to get started and you have the flexibility to take the control yourself if you wish so.

Language hosts

Responsible for running a Pulumi program. Consists of:

Language Executor
a binary that is used to launch the runtime (e.g., pulumi-language-dotnet.exe). Comes with the CLI. I installed Pulumi via Chocolatey so this is the location for this for me: C:\ProgramData\chocolatey\lib\pulumi\tools\Pulumi\bin
Language Runtime
Responsible for preparing your program to be executed. The runtime observes the execution of a program and detect resource registrations, communicates the registration request to the deployment engine. The runtime is distributed as a regular package. Here is the one for .NET on nuget, node on npm etc.

Deployment engine

The engine is responsible for achieving the desired state you have specified in your Pulumi program. Given the program from language host, it checks the local state, talks to resource providers and take all sorts of decisions to get you to the desired state. You interact with the engine with the CLI.

Stacks

A stack is an isolated, independently configurable instance of a Pulumi program. You use them to represent various environments like dev, stage and prod (I use them like this so far). Or you can represent feature branches like feature-x-dev, feature-y-dev.

Resource Providers

This is how you talk to your target cloud platforms like Azure, AWS and other SaaS platforms. Consists of two parts:

Resource Plugin Again, a binary that the deployment engine uses to manage a resource. You can manage plugins using pulumi plugin from the CLI You can find these in your user directory (for me it's at: C:\Users\sanja.pulumi\plugins - I'm on Windows machine)
SDK Just like the language runtime, available as packages. These determine the type of resources any provider can manage. For Azure, you have this nuget package for C#

Pulumi CLI

This is your gateway to the Pulumi world. You interact with Pulumi via CLI for all the operations you want to carry on.

Take Pulumi for a ride

Now we have got a fair understanding of how Pulumi works and what are various components involved. Let's change the gears and see it in action.

Setting up the environment

The first thing we need is to set the environment to work with Pulumi. Pulumi has a really good getting started guide. Here is the one for Azure specifically. So please follow the guide and install and configure your environment. What you should have after the setup is:

Pulumi CLI Installed
IDE of your choice (VS Code, VS etc.)
Azure Subscription
Pulumi Service Account (for state management) (https://pulumi.com)

I am setting the environment on my local Windows 10 machine. You can install Pulumi CLI in a docker container if you wish.

Hello Pulumi

Now having our environment setup, let's start with the quintessential Hello world application with Pulumi! And what's the Cloud equivalent of Hello World? Creating a resource group and a storage account - isn't it 😆 Let's do it!

Create a new project

Fire up your terminal and type the following command

pulumi new

This is an interactive command. You will be asked to choose the template first:

Next, set the project name, description, stack, location (you can simply enter to set recommended names or change the names).

Pulumi then goes ahead and initializes the project for you.

or you can simply pass the arguments at the time of creation:

pulumi new azure-csharp --stack dev --name hello-pulumi --description "Minimal C# Pulumi Program"

Let's see what Pulumi has created for us. Enter code . in your terminal to open up the VS Code.

Projects

A Pulumi project is any folder which contains a Pulumi.yaml file. A project specifies which runtime to use, which determines where to look for the program that should be executed during deployments. Supported runtimes are nodejs, python, dotnet, and go.

This is how my structure looks like:

My YAML files looks like this:

  name: hello-pulumi
  runtime: dotnet
  description: A minimal Azure C# Pulumi program

You also get a YAML file with your 'Stack' name that contains some configurations.

MyStack.cs file is actually the stack that contains our desired state which we want to achieve
Program.cs file is just an entry point to execute the stacks

using Pulumi;
using Pulumi.Azure.Core;
using Pulumi.Azure.Storage;

class MyStack : Stack
{
    public MyStack()
    {
        // Create an Azure Resource Group
        var resourceGroup = new ResourceGroup("resourceGroup");

        // Create an Azure Storage Account
        var storageAccount = new Account("storage", new AccountArgs
        {
            ResourceGroupName = resourceGroup.Name,
            AccountReplicationType = "LRS",
            AccountTier = "Standard"
        });

        // Export the connection string for the storage account
        this.ConnectionString = storageAccount.PrimaryConnectionString;
    }

    [Output]
    public Output<string> ConnectionString { get; set; }
}

Stacks inherit from Stack class.
In this program, we have defined a resource group and a storage account in that resource group. And finally we are exporting a connection string property (Output) once the storage account is provisioned. This will be output after the execution of the program.

Preview the changes

Once you are ready with the program, you can use the preview command

pulumi preview

This tells you what Pulumi is going to do when you actually deploy. This is a really good way to test the impact before actually deploying the program and allows for the faster feedback loop

As we discussed already, Pulumi offers a backend service for state management. If you follow the link (View Live) in the console, you will be able to see the state preview on the backend service.

It looks similar to what you already have on your console.

Let's deploy the changes now.

Deploy the Changes

pulumi up

By the way, pulumi up also does preview before and gives you the opportunity to go ahead of reject the deployment.

Go ahead and select 'yes' and let Pulumi deploy the changes.

As you can see, after successful deployment, you have a 'View Live' link taking you to the Pulumi backend service and 'ConnectionString' of the storage account as output.

Reviewing the Changes

Now that we have deployed the resource successfully let's check Pulumi's state by following the URL

Now, just move your cursor in Changes windows on Pulumi you will see few buttons appearing on the top right. Click 'diff' there. Here you will be able to see the log of what has been created/changed for you compared to the previous state.

Note, the name of the resource group here. In our Program, we simply specified the name as 'resourceGroup', this is not the real name, this way Pulumi generates a random unique name at the time of execution, that's why we get the random numbers after our specified name. If you want to specify the name of the resource you want, you will need to specify the 'name' property explicitly for that resource.

Verify the deployment

So far, Pulumi is telling us everything looks great but let's verify it in Azure. Navigate to Azure Portal and search for the resource group name you noted earlier.

So, it went fine after all. We have a resource group and a storage account within that resource group. All good

Update the program

Now, let's make a quick change by adding a tag in our Storage account resource. Add the following line after 'AccountTier' property in your storage account resource.

 Tags = {
          { "Environment", "Dev" }
        }

Back to your terminal, enter

pulumi up

Since we have made a change after deploying the program earlier, Pulumi already has a state of our infrastructure. When you do Pulumi up now, Pulumi will compare the state it has for our stack and will tell you what changes it has detected and going to update.

select 'yes' to deploy the change

As you can see, Pulumi successfully deployed our change. Follow the Pulumi link to preview the change. Notice, this URL is different to the previous one. You will be able to see all the updates that have made on your stack on the backend service.

Head back to the Azure portal and open the storage account again and see that our tag has been applied now.

Destroy

If at this stage, you want to delete the state, it's as simple as executing the following command

pulumi destroy

By this, we are telling Pulumi to just delete the stack.

If you head back to the Pulumi backend service and see the 'Activity' of your stack (dev) you will be able to see all the changes that you have made to your state

Summary

We discussed how Pulumi works, what are the core concepts while working with Pulumi. How to set up the environment and write your first Pulumi program. We also looked at the Pulumi's backend service (a free hosted state management service) to review the changes while you are working with your infrastructure.

Signing off

I hope this blog post helped you understand the core concepts involved while working with Pulumi and also gave your some idea of how it is to work with Pulumi. Give it a go, it's very easy to get started working with Pulumi. If you have any questions or feedback, feel free to drop the comment.

Cheers

Pulumi - What and Why?

Thu, 10 Sep 2020 00:00:00 GMT

I am learning Pulumi and I'll be sharing my journey along the way here in the form of blog posts and code samples on my GitHub. Join me in this journey if this topic interest you.

Disclaimer: I have hands-on experience working with Microsoft Azure Cloud platform not so much with others. So in here I'll be looking at Pulumi from Azure's lens and most of my references and examples will be with Azure.

Context

Infrastructure as Code (IaC) is the process of managing and provisioning computer data centres through machine-readable definition files, rather than physical hardware configuration or interactive configuration tools. It's the practice of treating your infrastructure like you would treat your code. Meaning, you can check in the definition files in a source control system, review it, version it and it produces the same environment every time it is used.

Infrastructure provisioning predates cloud. We used to provision our on-premises infrastructure before cloud became the thing and there are plenty of tools for that. With cloud comes the opportunity and challenge of managing large scale distributed systems, service based architectures and cloud native applications. If you are building apps on Cloud, you need to have automation and have the ability to quickly span up and tear down the environment in a short amount of time and with confidence. This is where the whole infrastructure as code approach shines. This gives you the superpowers to run any workload in cloud whilst taking advantage of core cloud tenants like speed, elasticity, cost-effectiveness.

Almost every cloud provider has a way to provision infrastructure. For this, they have their own domain specific language (DSL) and often written in JSON or YAML format - called templates. For Azure, its ARM (Azure Resource Manager), for Amazon its CloudFormation, Google its Cloud Development Manager etc. Now they are very powerful in a sense that you can declaratively provision your complex cloud resources without knowing the details of the underlying workings - the cloud platform takes in your JSON/YAML files and do the magic for you. This has helped the entire Industry a lot in adaption of cloud platforms, irrespective of which platform you are operating on. However, there are certain challenges associated with this.

With declarative approach:

You need to know and learn the DSL for each and every cloud platform you are working with or you need to type in YAML files (I know right !),
You rely on the (not so great) tooling around it
Templates get very difficult to manage and go out of control very quickly
Lack of local development loop. You have to actually call-in cloud APIs (actually deploy them) to be sure the templates you have authored are correct and will work.

Now take a notch up, let's say you are going to work across cloud platforms, i.e., n > 1. We can discuss whether it's a good idea or not but it's becoming a reality. Even if your application doesn't work in multi-cloud scenario but you are most likely using other SaaS applications or services for example, using Cloudflare for managing CDN for your web application running in Azure. This adds further challenges to the list previously mentioned:

You need to write infrastructure in different language for n number of cloud platforms you are working with
Each has its own set of associated tools you might need to deal with
Most likely, your infrastructure code will be placed in different locations not in a single coherent place (not necessarily but likely)
etc. etc.

...here enters Terraform !

Terraform solved this challenge of dealing with multi-cloud scenario. It's a very popular tool in IaC space today and I would say almost becoming the de facto standard when it comes to provisioning infrastructure in cloud these days. It's free and open source. However, Terraform introduces another DSL which is called HCL (HashiCorp Configuration Language). It is a declarative language where you define the resources you would like to provision in a cloud platform. It works cross platforms and you need to know only HCL to work against any platform of your choice. There is a great ecosystem and community around Terraform which is great. Even though Terraform is fantastic but now you need to understand yet another language.

...now we are entering the realm of Pulumi

Pulumi takes a step further and expands on this problem and provides us a way to really use any proper programming language of our choice and write the infrastructure provisioning code like we write any other code as developers and operations people.

What is Pulumi?

What Pulumi brings on the table is you can use the programming language of your choice to write the infrastructure for any cloud - "Any Code, Any Cloud"
Pulumi is a Modern Infrastructure as Code with which you can create, deploy and manage infrastructure on any cloud using familiar programming languages and tools.

As of this writing, Pulumi supports these programming languages: Go, Python, JavaScript, TypeScript, C#, F# and Visual Basic (.NET Core)

It's also free and open source.

Why Pulumi?

By leveraging familiar programming languages for infrastructure as code, Pulumi makes you more productive, and enables sharing and reuse of common patterns. A single delivery workflow across any cloud helps developers and operators work better together.

Some reasons I think Pulumi is great:

Programming model - You can write your infrastructure like you write your code in a programming language and tools of your choice This is HUGE. Having the flexibility to provision your infrastructure with your preferred programming language is a big productivity boost. On top of that you can bring in the same software development practices and principles like composition, abstraction, testability etc. for your infrastructure and use the favourite tools you are accustomed to, such as IDEs etc.
Familiar tools
IDEs, refactoring, testing tools, package management etc. Improves productivity from the get go
Configuration Management
Testability
With the power of programming language you can actually test your infrastructure like you would for your code. No one is stopping you to bring back the TDD from dead, if you will ;)
Faster feedback loop With programming language, testability and ability to preview the plans before you actually update/apply your changes comes very handy while writing the infrastructure code.
Multi cloud support
Pulumi allows you to provision resources in multiple clouds and SaaS services. Even though you will be writing separate code for targeting to different clouds (providers) but you can have all of them in a single solution/project at one place. On top of that, you can have your entire end-to-end across clouds and SaaS infrastructure in a single place that you can check in your source code repository in GitHub.
Governance
Besides providing infrastructure as code capabilities, Pulumi has introduced the support for Policy as code. You can write the policies for governance (which could be managed separately by security team, independent of the DevOps team) that can be executed while running Pulumi programs. It's not tied to cloud vendors (like Azure Policies), rather they are Pulumi specific but that can be run across clouds to enable same set of policies in multi-cloud scenarios.
Pulumi provides:
- Policy as code
- Secret Management
- Enforce standards
Adapting existing infrastructure You can bring in your terraform files to Pulumi (they provide a tool for this) or you can adapt existing environment from your already deployed resources in the cloud
State management
Besides providing the flexibility to completely manage the state ourselves (just like Terraform), Pulumi offers a free app.pulumi.com service that takes control of state management concerns for you.
Validating your code before you actually provision the resources
The ability to preview the changes you are making before you actually deploy the code helps in the faster feedback loop when you are writing the code.

Other tools in similar space

While looking at Pulumi I came across some other tools in this space that are doing similar things but there are still quite a few differences. I think Pulumi is far ahead of them - at least as of now.

Farmer
this lets you write the code in F# and generates ARM templates at the end
CDK for Terraform

What do I think about this

I am coming from a development background, so don't consider myself an Ops guy per se but let's be real. If you are working in cloud, the boundaries are getting blurred faster than we realise. If you are developing, the chances are you will be deploying the applications yourself. And having the mindset of operability of your applications you develop pushes you to build great software. It's a great learning experience in and of itself.

I think this space is going to get a lot of traction and will be adapted widely as we can already see the competitors getting into this space (referring to CDK). Even though it seems this is targeted more toward developers than operations people but the platform already support quite popular languages like Python and Go that are widely used in Ops world. Moreover, for Microsoft workloads, since we already have the support for .NET it's perhaps just a matter of time before PowerShell support is added. Another reason I see is the level of confidence this provides with the whole testability story that Pulumi enables. It allows us the flexibility to try out the infrastructure with confidence right from our local development machines and that only leads to the robust infrastructure. I'm excited about this if you ask me !

Wrapping up

Alright, this was a bit of an introduction to what Pulumi is and why it makes sense to me. As I'm learning more about Pulumi, I'll be sharing my learnings along the way. Thanks for sticking around.

Let me know if you have fiddled with Pulumi yet and how do you find it?

Until next time
Ciao

Passing AZ-220: Microsoft Azure IoT Developer Exam

Sat, 05 Sep 2020 00:00:00 GMT

I recently passed the AZ 220: Microsoft Azure IoT Developer exam and thought to share my impressions and some reference material that might of use if you are pursuing this exam.

The Exam

Candidates for this exam should have subject matter expertise developing cloud and edge components of an Azure IoT solution.

You implement designs for Azure IoT solutions including device topology, connectivity, debugging, and security, as well as solutions to manage, monitor, and transform IoT-related data pipelines. You also deploy Azure IoT Edge components and configure device networking on the edge.

This exam is targeted at Developers who will be building IoT solutions using Microsoft services and technologies.

You can read more about the exam here

Skills measured

Following are the skills you will be measured on. For detailed description, check this out

Implement the IoT solution infrastructure (15-20%)
Provision and manage devices (20-25%)
Implement Edge (15-20%)
Process and manage data (15-20%)
Monitor, troubleshoot, and optimize IoT solutions (15-20%)
Implement security (15-20%)

The exam frequently undergo changes so keep an eye on upcoming changes. You will find this on the exam page

Study Process

The study process varies from person to person, so there is no one approach. This is what I used for this exam.

Go through MS Learn Learning Paths and Modules
Hands-on Labs on GitHub for Azure IoT
Practice Exam (Mock)
Go back and cover the gaps
Repeat

Services

These are some of the Azure services that are relevant for this exam.

Azure IoT Hub
Device Provisioning Service
Azure IoT Central
Azure IoT Edge
Time Series Insights
Azure Stream Analytics
Azure Blob Storage
Power BI
Azure Event Hubs
Azure Service Bus
Azure Functions / Logic Apps
Azure Monitor / Log Analytics
Azure Cosmos DB

Study Material

This is what I used as study material for this exam.

Study Notes

A good starting point, gives you links to respective sections based on exam structure. Great time saver and you have links to all relevant areas and topics.

Please check this out.

Labs

Hands-on labs related to various topics for Azure IoT that are relevant for the exam.

MS Learn

There are few learning paths on MS Learn that are really good. I started off with these.

Microsoft Docs

Microsoft Docs are an invaluable resource anything Azure. There is tons of great documentation about any area and great in-depth knowledge in there. If you have found Study notes already, most of them will have links to MS Docs for relevant topics anyways. However, do dig into different topics as you are going through MS Learn paths or doing the labs to solidify the understanding. Here is the landing page for Azure IoT Docs

Courses on A Cloud Guru

I came to know about these courses quite late in my preparation so I didn't use them but they look good and could be a starting point for you, so do check them out.

Skills Assessment

As you are studying, fill-in this spreadsheet. It'll give you the overview of where you stand at any given point. You can use to track your progress and work on weak areas as you go about the studies.

Exam Experience - From Home

Due to COVID, most of us are stuck at our homes and to be honest, going to the physical centre sounds a bit risky at this time. Good thing is we have the possibility to give exams remotely and I've been using this for my last few exams. The overall experience has been positive (except once). So here are some tips:

Booking / Scheduling and Re-schedule

You can do this on Pearson VUE online. I have noticed, you can reschedule without any extra fees currently so it's quite flexible in that regards.

Set-up

Make sure you run the system test before booking online. You need to be sure the system you have is fit for the test. The utility you will be downloading from Pearson's website will test it for you.

Environment

One of the requirements is to have a clean space with closed door where no one can enter the room during the exam.
You will be monitored continuously via webcam
You won't be allowed to leave for any reason during the exam
Make sure you have a comfortable environment where you are giving the exam. Put the heater or fan (depending where you are)
You can keep water in a transparent glass or bottle
No phone at arms length. You can keep it far away (you might receive call from proctor during the exam if any issue arises)

Day of the exam

Start 15-30 minutes before the exam time so you have enough time to set up
You will be running the ID check (take a picture of your identity, your space and a selfie) and system check before the exam.
Then you simply wait for the proctor to start your exam.
They might call you (via phone or chat) to ask few questions or make some changes

Exam Tips & Tricks

Book the exam in advance (4-6) - give yourself a deadline and work toward it
Practice, Practice, Practice and some more
- Can't emphasize it enough. Do the hands-on labs, run the labs and demos. Try out. Explore. Get comfortable with the areas you are targeting.
- It's of no use to just pass the exam if you don't have any hands-on knowledge. Even if you haven't done any real life project and want to learn about any new area, do the labs and get the hands-on experience. There are plenty of resources available for any technology you want to get into. Put some hours in.
Know your Azure IoT
- Pricing Tiers of IoT Hub
- JSON structure of various components like Digital Twins, Module Twins, Device Deployment etc.
Clean your resources once you are done with the labs (to save cost)
MS Learn modules provide you the Sandbox environment - use them as it'll save you cost
Some resources are bit more expensive than others i.e., Azure Time Series Insights (make sure you delete them after you are done)
Take notes
When you are reading documentation or watching video content, take notes along the way. This will be helpful when you want to revise before the exam. You will save plenty of time with these notes.
Mix it a bit Studying for the exam can get boring, so make sure you mix it up a bit. If you are reading, add in bunch of videos along the way so you are consuming a mixed content during your preparation. You will loose the focus and end-up wasting a lot of time. Mix up various types of contents and hands-on on regular basis

My Exam and Impressions

I got 57 questions in my exam
- 42 questions (multiple choice types - radio / check boxes / drag & drop etc.)
- Set of questions (5 - can’t go back to review)
- 2 case studies (5 questions each)
- power
Time was not an issue. You get ~3 hours for this exam and even though there were quite a lot of questions but I easily managed the time
For me case studies were at the end which I find good since you end up spending quite a bit of time on case studies and I find them better to have toward the end (unlike other exams which started with a case study)
I can't list the exact questions I received but there were quite a number of questions with Azure CLI commands, IoT Edge and around security.
What helped me solidify the knowledge was doing hands-on (the labs that I have listed above and MS Learn modules), I did the more than once. Also for each services (like IoT Hub, IoT Central, Device Provisioning service),familiarize yourself in the Azure Portal. Just just checkout various options you have for these services.
Since I needed to provision resource a number of time (saving cost etc.), I tried to provision them from Portal, Azure CLI, Cloud Shell to just get the feel of different approaches)
I didn't spend too much time on PowerShell though.

Hope you find this guide helpful and wish you all the best.

Cheers

Capture your favourite quotes from physical books - Part 2

Tue, 25 Aug 2020 00:00:00 GMT

This is a two part series blogpost:

In my previous blogpost, we created the logic app where we could upload a picture to a folder in OneDrive and Computer Vision API detected the text from that image. Please read through that post if you haven't already before continuing this one. Let's take it a bit further in this blogpost.

Save detected text to a text file

Currently, there is no action for 'Append' operation on blob storage. This is bit of a bummer if you ask me! So we need to write a bit of a code to achieve this. I know I said, I want this to be easy but such is life, you got to to write some code somewhere you didn't anticipate. I'm going to pick up the Azure Function with C#, you can use whatever language you are comfortable with.

Create Azure Function app

You can pick any IDE you prefer, I'm going to work with Visual Studio (VSCode is amazing as well, if you prefer that)

I won't go into detail of how create and publish functions as it has already been written about. You can follow this link to create Azure Functions and Publish them to Azure. If you prefer VS Code check this out.

Once you have created the Functions project, Delete the 'Function1.cs' file and add a new Function (Right Click Project -> Add -> New azure function -> Select 'Azure Function' and name it AddQuotetoFile)
Select 'Http trigger' and press 'OK'

Moving on to the actual code. Here are some details:
The Function expects the following payload (request)

{
    "blobName": "testfile.txt",
    "containerName": "ocr-images",
    "text": "Here is the quote from the book"
}

blobName: the name of the blob that we need to append the text to (if it - doesn't exist already, it'll be created the first time)
containerName: the blob storage container where the file should exist
text: the actual text we want to write to the file

In this function, when it's triggered, we want to:

Parse the HTTP request body
Make sure we have all the information we need
Grab reference to the blob container (create if it doesn't exist)
Grab reference to the blob (create if it doesn't exist)
Append the text at the end of the file

This is how my main function looks like. Please check out my GitHub repository for the entire solution.

public static class AddQuotetoFile
    {
        [FunctionName("AddQuotetoFile")]
        public static async Task<IActionResult> Run(
            [HttpTrigger(AuthorizationLevel.Function,"post", Route = null)] HttpRequest req,
            ILogger log)
        {
            log.LogInformation("C# HTTP trigger function processed a request.");

            // Expects JSON body:
            // {
            //      "blobName": "[Blob name here]",
            //      "containerName": "[Container name here]",
            //      "text": "\"Column A\","Column B\",\\n,\"A1\",\"B1\",\\n,\"A2\",\"B2\""
            // }

            var connectionString = System.Environment.GetEnvironmentVariable("StorageConnectionString");

            // StorageConnectionString configured as app setting.

            string requestBody = await new StreamReader(req.Body).ReadToEndAsync();
            dynamic data = JsonConvert.DeserializeObject(requestBody);

            var blobName = data?.blobName;
            var containerName = data?.containerName;
            var text = data?.text;

            if (blobName == null)
            {
                return new BadRequestObjectResult("blobName is required.");
            }

            if (containerName == null)
            {
                return new BadRequestObjectResult("containerName is required.");
            }

            if (text == null)
            {
                return new BadRequestObjectResult("text is required.");
            }

            var storageAccount = CloudStorageAccount.Parse(connectionString);
            var blobClient = storageAccount.CreateCloudBlobClient();
            CloudBlobContainer container = blobClient.GetContainerReference(containerName.ToString());

            await container.CreateIfNotExistsAsync();

            
            var appendBlob = container.GetAppendBlobReference(blobName.ToString());

            if ((await appendBlob.ExistsAsync()) == false)
            {
                await appendBlob.CreateOrReplaceAsync();
            }

            StringBuilder lineToAdd = new StringBuilder();
            lineToAdd.AppendLine(text.ToString());
            await appendBlob.AppendTextAsync(lineToAdd.ToString());

            return new OkResult();
        }
    }

Notice in the code snippet, we are referencing 'StorageConnectionString' environment variable. This is the connection string of a storage account where the file is being created/appended. For this, we need to create a storage account.

Head over to the portal
Create a resource for Storage Account
Press 'Review + create' to create the account
Head over to the resource once it's created
Select 'Access keys' under Settings
Copy the Key1 connection string
Back to Visual Studio, open your 'local.settings.json' file and set a property 'StorageConnectionString' under Values

{
  "IsEncrypted": false,
  "Values": {
    "AzureWebJobsStorage": "UseDevelopmentStorage=true",
    "FUNCTIONS_WORKER_RUNTIME": "dotnet",
    "StorageConnectionString": "<your connection string goes here>"
  }
}

Make sure the you can Build the function correctly.

At this point, if you want to test the function locally, hit F5 to debug the solution, you can make a HTTP request via Postman (or any other tools) to ensure the function is working as expected. Drop a file in a container in the storage account you created and construct the HTTP request and hit your local function endpoint (you can use simulator to pick the file from instead of using Storage Account in Azure, if you have it setup).

Once you have established the function works correctly, it's time to publish the function. We all know "Friends don't let friend right-click and publish" - but it's OK for this demonstration, so don't feel bad and go ahead - yes Right click the solution and hit Publish. Follow the links that I mentioned earlier (here and here) for the instructions on how to run locally and publish from Visual Studio and Visual Studio Code.

Once the publishing is successful, we would like to add the application setting.
Navigate to the Azure portal and open the function app resource that we just published
Head over to 'Configuration' under Settings
Add a new application setting
- Name: StorageConnectionString
- Value: "the connection string of storage account you put in local.settings.json file earlier"
- Make sure to click save to persist the setting, otherwise you will loose it.

Now we are ready with the setup. You can test this at point via Postman as you did earlier. The only change would be to pick up the URL of the function from Azure instead of localhost. Still in your Function in the Portal -> Select 'Functions' -> Select your function ('AddQuotetoFile) -> click on 'Get Function Url' and copy the URL.

Update the logic app to trigger Azure Function

Head over to the Azure Portal, Navigate to the Logic App we created in Part 1. Select 'Logic app designer' under 'Development tools'
Click '+ New step' to add an action
Search 'Compose' -> Select 'Data Operations' and then 'Compose' action.

Under Inputs, add this JSON

{
  "blobName": "quotes.txt",
  "containerName": "quotescontainer",
  "text": @{body('Optical_Character_Recognition_(OCR)_to_Text')?['text']}
}

Add another action
Search for 'Azure Functions' -> Select 'Azure Functions' and then 'AppendQuoteToFile' action
Set 'Request Body' to compose outputs
Save the Logic App

We are almost done, time to test it out now. Upload another image in your OneDrive folder 'ocr-images' and watch the run history again. Here is my successful run:

Let's open up the storage account to see if the text was written successfully.

Select the storage account -> Containers -> quotescontainer
click on the file 'quotes.txt' (notice the Blob type is 'Append blob')
From the pop-up press 'Download' and open it in any editor and inspect the contents

I uploaded yet another file in my OneDrive. Download the quotes.txt file again and look at the contents. You will see another line has been appended.

Here are all the resources we ended up creating for this solution

We are finally done! Now when you take a picture of any quote and upload it to your OneDrive, Logic app will be triggered and appends the quote into a text file in your blob storage.

In case you followed along and provisioned the resources and you don't intend to use the solution, you can delete the resource group here.

This turned out to be a much longer blogpost than I anticipated, so thanks for tagging along.
Let me know if you found this useful and if you have any suggestions to improve this further.

Cheers

Capture your favourite quotes from physical books

Mon, 24 Aug 2020 00:00:00 GMT

This is a two part series blogpost:

I like reading books. Though I love technology but certain books are best suited for physical form and I'm quite open about it - whatever works. One of the good things about digital books is they make it quite trivial to highlight or take notes of our favourite quotes or sections. When it comes to physical books, there are all kind of (manual) techniques you can use to take notes. While underlying few quotes in a book I was reading the other day, I thought wouldn't it be nice if I take a picture and the quote is written to a text file somewhere as a collection for me which I can refer to later on? Of course, I didn't just put down the book right away since I was enjoying it so I carried on - I know not so easy to do.

Once done with the reading, I set out to create a solution for this. I work with Microsoft Azure in my day to day job so it was an obvious choice for me. I went about picking up the services that could help me easily put different pieces together to solve the challenge at hand.

This is what I came up with.

So the flow is: I will take a picture of any quote with my phone, upload it to a certain folder in my OneDrive. The Logic app gets triggered if there is a new file in that folder, reads the contents of the file, passes that image to the Cognitive API (OCR to Text) that returns the detected text, we then call an azure function that appends the quote to a text file in my blob storage.

Rationale

OneDrive (Personal): I already use it, it's my private drive, have secured access from anywhere and from my device
Logic Apps: Helps me orchestrate the flow without writing much code and I can use built-in connectors to access OneDrive, Cognitive services, blob storage etc.
Computer Vision API: Makes it almost trivial to detect text from the images and works quite well with Logic Apps
Azure Functions: Though I shouldn't have used this but this is to append the text in blob storage since there is no built-in connector for this currently.
Blob Storage: I could potentially store this file anywhere else but storage account is quite handy in this scenario (supports append blobs natively).

With the high level architecture in place, let's dig into the implementation.

Create Logic App

Head over to the Azure Portal and create a logic app.
Once the resource is created, go to that resource.
Select 'Blank Logic App' from templates
Search for 'OneDrive' connector and select 'When a file is created' from Triggers
Sign in to your OneDrive account and select the folder where you will be placing your images from mobile.
Set the frequency at which you want OneDrive connector to look for new files (during development, select smaller window and when you are ready to use it, increase it so you don't end-up polling OneDrive too often)
Click 'New Step' to add the next action.
Search for 'Cognitive' and select 'Computer Vision API' and then search for 'ocr'.
Select 'Optical Character Recognition (OCR) to Text' from the list
To fill this information we need a Computer vision resource.

Create Computer Vision resource

Open another tab/window and go to Azure Portal (portal.azure.com)
Create a new resource of type 'Computer Vision'. Select 'F0' pricing tier (it's free - though you can have only one free resource in a subscription)
Once the resource is created, navigate to that resource (by selecting 'Go to resource' or search for it from the search bar at the top)
Head over to 'Keys and Endpoint' under 'Resource Management' section
Copy 'Key1' and 'Endpoint' (in notepad or anywhere else you prefer)

Update Logic App with Computer Vision Action

Head back to the Logic App now
Enter connection name 'computervisionconnection'
Enter value of 'Key1' from computer vision API in 'Account Key' field
Set 'Site Url' to the 'Endpoint' you coped from computer vision
Click 'Create' button to create the connection
For 'Image Source' - select 'Image Content' from the dropdown.
For 'Image Content' - set it to File Content from previous step (by selecting the dynamic content. Put a cursor in - Image Content field and a pop up will show up, select the value from there)
This is how the logic app looks so far
At this point, we have a logic app that will be triggered if there is a new file in OneDrive under '/ocr-images' folder and the text is translated using Computer Vision API. It's time to test

Test the app

Save the app. Upload a picture of some text in your folder in OneDrive. Check the run history of the logic app and select the last run.
I have uploaded this picture
Here is how it looks for me.

Check the text in 'Body' and you will see that it matches the text we had in the image. That's pretty rad, isn't it?

We are pretty much done here. But I would like to take this further. I would like the text to be stored in a text file that sits in my blob storage. I want the text to be appended every time the logic app runs so I have my quotes in one file on new line. This will be covered in part II of the series, so stay tuned.

Cheers

Migrating from WordPress to Wyam

Sat, 05 Oct 2019 00:00:00 GMT

I had been hosting my blog on SiteGround for a number of years now. It's been good so far, no complaints but I had been wanting to try out the static content generation engines for a while. One major reason for this is that my content is mostly static and don't really require a server side for anything as such (ok, few things but nothing that can't be achieved on client-side).

I had looked at few options in the past but never really set out to do this. What pushed me to make this move was that, during last month, I got an email that some malicious code has been detected on my website! SiteGround did a really good job to make my website unavailable and notified me so that I could investigate, clean it up and put it back. It was a fairly smooth process but it got me thinking, this shouldn't have happened at first place. I don't really need the overhead of dealing with server side code at all. I could have drastically reduced the attack surface just by serving the static content. So, this was the tipping point for me.

I have been following static content generation solutions and was aware of this space and few options like Jekyll & Wyam etc. I decided to go with Wyam for now (it's built on top of .NET Core) but any other engine will do. My motivation to go with Wyam was influenced by a number of people I follow in the tech community and the fact it's running on .NET which, for me, is good opportunity to understand and learn how things are working behind the scenes to make such application work.

This is how my migration process looked like:

Installed Jekyll Exporter plugin on my wordpress site and exported the content in markdown format
Pretty much followed this blog to do the heavy lifting, so won't repeat myself again as it has already been written quite a lot. Here is another good blogpost
Some decisions that I made though:
- Went with CleanBlog theme but made some changes (css, fonts etc.) to suit the design I liked.
- The exported content is not really proper markdown but I am OK with it. It would have taken significant effort to clean it up otherwise.
- Decided to put images along with the content for now. I will look further if there is any better approach like hosting images into Azure Blob storage.
- Even though I had disqus comment system enabled on my wordpress, I encountered a bit of a trouble getting it right because the out-of-the-box wyam URLs were different than my previous and during testing I ended up creating different disqus threads for my posts and it messed it up. I needed to remove those threads from disqus to be able to re-use with my previous URLs. You can read more about this here
- Configured the URLs to be same as my previous URLs (includes day in the post path besides the default year and month). I found this way simpler rather than doing the redirections etc.
- Added support for Google Analytics (might get rid of this going forward with some other solutions but good for now)

Release Management:

I have hosted the source code (aka, markdown pages and assets) on GitHub
Setup Netlify account to host the site (I could have hosted it directly on GitHub but wanted to try out Netlify as I was hearing so many good things about it lately,rightfully so. It was a delightful experience with Netlify.)
I setup Azure DevOps pipeline (via yml) to push the content to Netlify
I found this a lot easier and simpler without going to any other approach (Cake or PS). I'm tempted to use Github Actions for this part however.
Whenever I push anything into my 'master' branch, the pipeline is triggered and content is pushed over to Netlify.
- One nice thing about Netlify is that, it allows you to preview your changes before you want to Publish it, which is nice (there are other options to create proper staging environment etc. but for me this was sufficient)

Here is my new tech stack:

Wyam - Static Content Generator
GitHub - hosting the content
Netlify - hosting the blog
Cloudflare - Caching, DNS and Certificate management (via Lets Encrypt)
Azure DevOps - CI/CD
Google Analytics - Traffic Analysis

That's about it. With this setup, After pushing everything I redirected my domain from SiteGround to Netlify site (from cloudflare) and bam new blog is alive!

Now, I want to list some things that I like, did not like and what I will add / improve going forward.

Things I like:

The entire blog is served as static resource (html, images, css etc.), nothing is running server side (as of now at least)
Wyam is pretty awesome and seem to work really well. I do look forward to learning more about this.
I am able to write my posts totally in markdown on my local machine (without any internet connectivity with amazing VSCode)
Site is much faster!

Things I am less satisfied with:

Wordpress took care of a lot of things out-of-the-box for me such as search, image compression, contact form and various other plugins (pretty for any need you can think of) and I will have to add a number of these features to my site which I find important. But I was fully aware of these trade-offs.
I don't understand Wyam 100% yet as there is some learning curve and Wyam's own tooling (no intellisense for example) makes it a bit difficult to compose config file.
I need to think of image compression or use better solution to reduce the size of served content.
Had some issues with setting up Images compression with Wyam (apparently there are some issues around this)

Things I will add/improve going forward:

Add Search capability on my site as I think it's very important feature (and one that I use quite often no matter where I'm as a user)
Better support for serving smaller images (or even replace the ones that I already have with smaller size)
Contact form
Might replace Disqus with GitHub issues (found out this Utterance)

Another thing I found interesting is site performance. Up until now, I didn't pay much attention to this aspect since it's not a heavy site but as I recently moved to Australia, I started noticing how slow internet speed affects your mood :wink:, I decided to improve this aspect of my site. By following the approach for my new blog (without doing much optimization, I already gained a huge improvement), see it yourself:

This is my site on wordpress earlier (see both mobile and desktop speeds):

And here is my new blog (again, mobile vs desktop):

That's quite impressive already!!. I have yet to optimize images (which should push me even further). I can make some optimizations in time to improve it even further. I used Google's PageSpeed Insights to measure this.

That's it for now. I will keep posting my learnings and finding as I go along.

Feel free to drop any suggestions if you have been the same path already, I would very much appreciate that.

Cheers

Microsoft OpenHack DevOps - Stockholm

Mon, 11 Feb 2019 00:00:00 GMT

Last week I participated in OpenHack DevOps organized by Microsoft in Stockholm. It was a three day event from 5th - 7th February 2019. The hackathon was focused on Azure DevOps. The format of the event was of-course like a hackathon.

Participants were divided into teams (4-6 each). The content was set as challenges (total 9). Every team (or two) had a coach - someone from Microsoft - who was helping and guiding during the challenges. We, as teams, were supposed to find our way out to solve the challenges. There was no one way, we were free to take our decisions and paths as deemed fit.

Here is the schedule and agenda for the event if you are curious: Microsoft OpenHack DevOps

What I really liked about this event was that we were focused on a task at hand. It was a hands-on rather than attending any tech talk about a specific topic. The tasks were set, challenges were well organized, the environment was prepared, code was almost prepared (with some changes) so that we can focus on learning how we can use Azure DevOps as a tool to ensure zero down-time for production ready application. Kubernetes was chosen as an orchestration framework.

Microsoft OpenHack is a developer focused event where a wide variety of participants (Open) learn through hands-on experimentation (Hack) using challenges based on real-world customer engagements designed to mimic the developer journey.

For every challenge the links to documentation and resources were provided to understand relevant topics and areas at hand.

Besides the actual work, it was a great opportunity to network and discuss broader topics with fellow participants and Microsoft employees. They also had what they called the envisioning sessions which were basically to discuss with Microsoft guys one-on-one the challenges one is facing at his or her work.

Overall, I think it was a great learning experience with great focus on getting things done. I will definitely keep an eye on such events in the future. By the way, if you happen to be in Stockholm in April there is an upcoming Microsoft Ignite Tour that you must checkout.

Cheers

Using Postman for Integration Tests

Tue, 11 Dec 2018 00:00:00 GMT

In my recent project, I'm providing some integration services (APIs) that are used by the web application to fetch data from source systems. This is a legacy project that I inherited. To be honest, this was a PoC and we all know how PoCs are developed and very often continue to be used in production. That is why I have been rewriting/refactoring this codebase (I even tweeted about this a while ago) in every sprint to improve the code quality and cleaning up a lot of code. Since I don't have a proper code coverage for all the source code, I'm always concerned when I'm continuously developing and refactoring this codebase. I'm often concerned (rightfully so) that I'll break my API and will eventually bring down the system due to some stupid mistake. But until I can have a good enough code coverage I want to be confident that I'm not breaking my APIs whenever I'm pushing a new release to production. Now I've taken quite a lot of steps over past few months (from having proper CI/CD pipeline to having slots support to have the availability for new releases etc) to clean up the codebase and continue to have my services up-and-running, one thing I'm particularly excited about is Postman.

If you are involved in RESTful API development, chances are you are already aware of Postman or atleast have heard of it. But in case this is the first time you heard about the tool, I'll give you a very quick intro about it.

"Postman mkes API Development Simple" Postman website

Postman is a powerful HTTP client for testing web services. It makes API development faster, easier and better. You can read more about it on their official website

I've been using postman during my development (testing APIs etc) but for last few weeks, I've started to write integration tests for my APIs so that whenever I'm making any changes or creating a new release (for any environment), I can quickly run the collection and make sure everything is still functioning as expected.

This is not a replacement for having a proper unit test (with good enough code coverage) but I find it very useful to quickly verify that I haven't broken the core functionality of my solution.

Here are some features I'm using with Postman for my integration tests

Test automation
- Collections
  - One big collection to run tests for all services in one go
- Separate collections for each service (API) with calls of each operation that API exposes/supports
- Extended tests to cover different scenarios
Debugging
- Pre-script and tests for individual calls
Environment variables - to have the possibility to test the APIs towards any environment that I want to test

Even though I've used quite a few features so far but it doesn't end here. Here is what I have planned to take advantage of other features as I go along.

Integrating postman in my CI/CD (newman) so that I can auto swap my slots for the web application if everything is green
Mocking
Continue to extend my tests cases to increase coverage

How are you using Postman in your projects? Have you found anything intersting/useful that you would like to share?

Cheers

Accessing WCF Service via Azure Service Bus Relay with .NET Core

Wed, 28 Nov 2018 00:00:00 GMT

Check out my GitHub repository if you want to dig into the source code directly.

WCF (Windows Communication Foundation) has served us for a long time when it comes to talking to many LOB systems (SAP etc.). You might have love or hate (though if I have to guess, it would be on hate side) relationship with WCF but it works and at times we don't have any other choice. If you happen to be in a situation where you are talking to legacy systems via WCF, you most likely have a built an API layer on top of it to serve your clients.

I was recently looking to consume WCF services using .NET Core (with the intention that I can migrate my web application to .NET Core). Very quickly I came up to this GitHub repository which was awesome. However, this .NET Core implementation is not in parity with the full framework yet. I tested it and it worked just fine with standard bindings (http etc). Shane Boyer has a good blog post on this topic if you want to see how to use this.

As far as my needs (this particular scenario) are concerned, I couldn't get it to work for my setup. This is how the setup looks for me.

The web application (hosted in Azure Web App) talks to WCF Service (running on-premises, hosted in IIS) via Azure Service Bus Relay (in Azure)

So basically, there is no equivalent to basicHttpRelayBinding in .NET Core implementation yet and this is something my WCF service has to enable communication via Azure Service Bus Relay.

After discussing with a friend of mine, I got a suggestion to try it out with SOAP instead and it seems to work fine! I know it sounds a bit ironic that on the one hand I'm moving to the latest (and greatest) framework for my web application but at the same time, I'm going one step back and using SOAP protocol to make it work - well such is life. I would very much appreciate if anyone can point me to the better solution. but until then, let's continue forward.

So, in this blog post, I will go through how did I manage to call WCF service from my .NET Core Web Application. Again, here is the entire source code

So this is what I'm going to do:

Create Azure Service Bus Relay (read here how to create this)
Create a WCF Service (with Azure Service Bus Relay Binding) and publish this locally to run it on IIS
ASP.NET Core Web Application calling to WCF Service using HTTP Post SOAP request.

Follow the link I have provided above to create the service bus relay. Let's start with WCF Service. Here is the sample WCF project I build for this demo. It's a very basic WCF service which has Azure Service Bus Relay binding besides standard HTTP binding (check out the web.config for binding details). I have deployed this in IIS on my local machine. You can clone the repository and add the details of your relay if you want to try it out yourself. It exposes the following contract, which I'm interested in.

[ServiceContract]
    public interface ICustomerService
    {
        [OperationContract]
        string GetCustomerData(string customerId);
    }

When you browse this service from IIS you should see the service like this. At this point, a Service Bus WCF listener should also be registered in your namespace (you can check it in Azure Portal)

WCF Service deployed in IIS with Azure Service Relay Binding

WCF Relay Registered from WCF Service

Now, let's start with the web application. I have created a vanilla ASP.NET Core Web API project (2.1). You can grab the source code from here (and clone it if you want to try it out yourself). Make sure you update settings in appsettings.json file for Azure Service Bus Relay. If you wish you publish this to Azure Web app - add these settings as Application Settings

After creating the project, I added following NuGet package

Nuget Packages:

Microsoft.Azure.ServiceBus

In ValuesController, I have created a GetCustomerData method that takes in customerId as a string. In here, I'll make a call to WCF Service and return the response I receive from the service (the WCF service returns a dummy response)

[HttpGet("{customerId}")]
public async Task<string> GetCustomerData(string customerId)

Then moving forwards, I generate the token for Azur Service Bus Relay:

Generating Token for Service Bus Relay

With this information, I construct the HTTP Post request like this:

We are basically making an HTTP Post Request to the Service Bus Relay endpoint over SOAP. For this, we need to set up a few headers

ServiceBusAuthorization - Relay Token retrieved earlier
SOAPAction - The Operation you want to call from WCF Service
Accept header - text/xml

Lastly, we need to set the body for the request - since this is a SOAP call, we need a SOAP message.

SOAP message body

Press F5 and run the web application, pass in cusotmerId parameter and you should get a response from the service:

Successful response from WCF Service

Have you tried to consume WCF Service from .NET Core application? How did it go for you? I would love to hear from you and would appreciate if you could suggest a better way of doing this as of today? I guess I don't need to tell you that I don't like using SOAP :)

Cheers.

Accessing Azure Analysis Services Models using .NET Core

Mon, 12 Nov 2018 00:00:00 GMT

Update (2020-07-25): Microsoft recently announced the .NET Core support for Azure Analysis Services client libraries in preview (AMO and ADOMD.NET). You can use these packages instead of the Unofficial packages which I have based this blogpost on. These packages are still in preview though, so you can't (or shoudn't) use these in your production environment. I have added another console application project in my project which you can checkout here. I just updated the reference to the nuget packge in .csproj file, nothing else. These libraries should work similar to the full .NET framework ones.

Azure Analysis Services is a fully managed platform as a service (PaaS) that provides enterprise-grade data models in the cloud. Use advanced mashup and modeling features to combine data from multiple data sources, define metrics, and secure your data in a single, trusted tabular semantic data model. The data model provides an easier and faster way for users to browse massive amounts of data for ad-hoc data analysis.
Refer to Microsoft official documentation to read more about Azure Analysis Services

Programming against Analysis services is nothing new and we have been doing it for a long time with the full .NET framework, the most common approach is using ADOMD.Net. In this blog post, I will go through the process of getting the same task done with .NET Core. For this sample, I'm using .NET Core 2.1.

Look at my GitHub repository for the entire source code for this blogpost

The important thing to note here is that there is no official nuget package from Microsoft for ADOMD.NET yet, but I found an unofficial package here (Unofficial.Microsoft.AnalysisServices.AdomdClientNetCore) and it seems to work for my quick test (you have to make a call if you want to use it in production or not). I couldn't find any official word on this anywhere I looked for. Besides this nuget package for .net core rest of the stuff should work same in full framework (with official nuget for ADOMD.NET)

I have divided this into several steps so that it is easy to follow. So let's get started!

Step 1: Create Azure Analysis Service resource

The very first thing we need is the Analysis Server and model in Azure. Follow this quick starter to create the server.

Next is to create a model which we will use to query. You can create a model with sample data (adventure works) right from within your Analysis Server.

Click 'Manage' in the blade and click 'New Model'. Select 'Sample data' from the drop down and press 'Add'. It should add the model for you.

Create a model Analysis Services

Model successfully created

Step 2: Create App Service Principal

There are many ways to access analysis services. Simplest is using a connection string that has usrename and password. But this is not recommended approach and works only with full .Net framework but not .Net Core (I was pointed by bdebaere in his GitHub respo regarding this), so we want to authenticate with other OAuth flows. For this post, we will use token-based authentication. For this we will need an app principal (or Azure AD App)

Sign in to Azure Portal.
Navigate to Azure Active Directory -> App Registrations and Click New application registration.
Register an app with the following settings:
- Name: any name
- Application type: Web app/API,
- Sign-on URL: https://westeurope.asazure.windows.net (Not really important here, you can provide any valid url).
Once the app is created, navigate to 'Keys' and add a new key
- provide the description and select duration and press Save button
- after that you will be able to see the key it will appear only once so take note of this key and we will use this later on
Also take note of the Application Id from the main page of the application

Setting access key for Azure AD App

Step 3: Assign your user as Service Admin in order to connect from SSMS

Registering an app is not enough. We need to assign access to this app on Analysis Service Model (adventureworks model that we created in previous step). In order to give this access, we will need SQL Server Management Studio.

Before we could use that, we need a way to connect to this analysis services instance via SSMS. For this, we need to set up our account as Service Admin. Navigate to the Analysis Services resource that we created in the first step. Click 'Analysis Services Admin'. Normally your subscription account is set as the admin (this is what I will be using) but you are free to set up any account you wish appropriate.

Setting Sevice Admin for Analysis Service

Step 4: Grant permissions to app principal on the model

Connect to Analysis Service using SSMS 2017 with your account that you assigned as Service Admin in the previous step
- You will need the Server name (from Step 1)
Select the database model and click on Roles or add a new Role
- Choose any name
- Select 'Read' database permission for the role
Add the Service principal to any role in below format (search for the app name)

Connect to Azure Analysis Server with Service Admin account

Adding App principal as a member in newly defined role

This will add user with following convention: app:<appid>@<tenantid>

appid: is the application id for your app you created in the Step 2.

tenantid - is the id of your subscription (you can find this in Properties of your Azure Active Directory)

This didn't work for me when I tried to use Azure subscription with my personal account (hotmail) so I had to use my company account subscription to make this work.

Step 5: Write the code to access data

Now we are all set up write our code that reads from the Model. Please refer to the entire source code in my GitHub respository

Important method here GetAccessToken. I'm using ADAL.Net (nuget: Microsoft.IdentityModel.Clients.ActiveDirectory) to grab the token for the service principal from Azure AD.

Method to acquire token from Azure AD to access analysis services

Once we have the token, we are good to access data from the model. Here I'm using the unofficial NuGet package for ADOMD.NET that I mentioned previously. The correct Connection String format is:

“Provider=MSOLAP;Data Source=<url of the Azure Analysis Server>;Initial Catalog=<modelname>;User ID=;Password=<access token here>;Persist Security Info=True;Impersonation Level=Impersonate“

User ID is left empty and Password is the access token which we get from Azure AD.

Method to read data from Analysis Services Model adventureworks

If you run this, you will see the output in the console

Final output of the program

Have you tried to work with Azure Analysis services in .NET Core? How was your experience? I would be very interested in listening to your experience and challenges.

Cheers

Are you keeping your secrets secret?

Thu, 25 Oct 2018 00:00:00 GMT

As developers, we are all guilty of leaking sensitive information of our applications and systems more than we would perhaps like to admit. Don't get me wrong, I'm not talking about breaking the NDAs with our clients but about those little connection strings to our databases and keys to our storage accounts where we hold all our information that we want to protect from outside world. As developers, we understand that this information is sensitive and still we check-in those along with the rest of our code base which eventually ends up in our version control systems that are hosted in the cloud somewhere. And we firmly believe that this is going to be fine. This is akin to watching a dreadful accident on NEWS and saying this can't happen to us.

"The Cloud" is all beautiful and powerful but with power comes responsibility. Among all people, we developers should be well aware and feel responsible & accountable for handling the sensitive information of our applications. We must ensure that it doesn't leak under any circumstances which can jeopardize our systems (and also our positions). And believe it or not cloud has made these things easier and we have all the tools we need to make it happen. It is not complicated anymore, you just need to be a bit thoughtful and set up the habit to do this when you are starting your project (or even better put these things if you end up in a project where this is lacking).

Ok, enough talking let's get to the meat of this post. In this blog post, I will create a Web Application (hosted in Microsoft Azure) which lists URIs of blobs from a container in Storage Account (Microsoft Azure). For the sake of simplicity, I'm creating the container and a dummy blob at the runtime. The Web app is a standard ASP.Net Core web app. I will present two implementations of this scenario.

The first implementation would be with the connection string of a Storage account stored in Application Settings of the web application directly. In the second part, I will move that connection string from the web application and keep it somewhere safe and will fetch information from the Storage account as earlier. Both the applications are deployed using ARM Templates and completely automated (You can create these resources manually if you so wish).

Part I

Here is the source code for this part. I will refer to some bits and pieces of this below:

Here is how the setup looks like:

Web App reading from Storage Account directly using Connection String

If you look at the source code (here), in my WebApWithSecrets solution, I have two projects:

Solution structure - WebAppWithSecrets

deployment project contains the ARM template for provisioning the resources. The ARM Template does the following

provisions a Storage Account
provisions the web application
adds 'StorageAccountConnectionString' as a Connection String in the Web Application

You have instructions to deploy this template in GitHub repository at the above URL (if you want to try it out yourself). Once the resources are provisioned, this is how my resource group looks like when I see it in Azure Portal.

Three resources provisioned by ARM template in Azure subscription

If I navigate to my Web Application and look at the Application Settings, this is what I see: a connection string with the name 'StorageAccountConnectionString'

Connection String set in Web Application after deployment

Once the resources are provisioned and in place, we can push the code to our web application. You can push this in a variety of ways but for now, I will just right-click and publish (WebAppsWithSecrets project (don't do this in production). Once the code is pushed successfully, the web application will open up in my browser (otherwise navigate to the web application). This is how it looks for me (the red box is my custom code that lists the URIs):

At this point, the app is fully functional and there is nothing wrong here. If I navigate to appsettings.json file in my source code, I will find this: a connection string to my storage account which my code picks and fetches content from Storage Account (you will find the code to render these URIs in Index.cshtml.cs file)

Connection String in appsettings.json in the solution

I have this details not only to me but this is now checked-in to my source control.

We can all argue that it is not that bad after all since this is hosted in GitHub (or Azure DevOps or whatever the version control system you are using). These are secure systems and only our team members have access to them - so why bother?

This all holds true until someone gets hold of your GitHub account or you haven't configured the permissions correctly and someone who shouldn't have the detail might get access. Put it simply, we are increasing the attack surface and we might have to deal with some unwanted situations later on. The good news is we don't necessary have to do it this way, there is a better alternative which I will demonstrate in next part.

Part II

Here is the source code for this part. I will refer to some bits and pieces of this below:

The alternative approach could be that we keep such sensitive information in some sort of secured place or vault which is encrypted and secure which we can rely on and our application simply asks for this information from that vault without ever knowing the details itself. We sort of want to delegate the responsibility of handling sensitive information of our application to some other party without us worrying about the implications. By doing this we not only simplify our lives but also gain much more benefits like it would be better to govern these secrets, we would be able to update the keys and connection strings and certificates without changing anything in our application. The good news is Microsoft Azure has exactly such offer called 'Azure KeyVault'. You can read about this more here.

This is how the redesign of our application looks like after introducing the Azure KeyVault:

Solution design after introducing Azure Key Vault

The Web Application has information about the KeyVault only and delegates the responsibility of retrieving ConnectionString to Storage Account to the KeyVault.

If you look at the source code (here). In WebApWithourSecrets solution, I have two projects, same as before.

I have updated my ARM template for this. The template now does the following:

provisions a Storage Account
provisions the KeyVault
provisions the web application
adds 'StorageAccountConnectionString' as a Secret in the KeyVault
sets the name of the KeyVault in application settings

There is one catch, however. You might ask, wait for a second here! Fine, I moved my storage account's connection string to the KeyVault but I still need details of the vault in my web application to read anything from it. Though I moved the secret of storage account my web application now holds even more sensitive information - all the secrets !! I feel you, I'm with you. But don't worry, this is a solved problem. Microsoft quickly realized this and provided a very nice solution for this very problem. They introduced something called 'Managed Service Identity' aka MSI (now being renamed to just Managed identities) a while back. You can read more about this here but in short, it abstracts this detail in such a way that you don't need to keep any ClientId/ClientSecret or Key of the vault in your web application. You simply enable Managed Identity in your web application (it creates an app principle which gets permissions to read from the Vault). Here is a really good premier of MSI if you want to get a deeper understanding of the feature.

If you navigate to Program.cs file in the solution, here is code that hooks up the KeyVault into our application (BuildWebHost method):

public static IWebHost BuildWebHost(string[] args) =>
     WebHost.CreateDefaultBuilder(args)
     //this is where KeyVault magic happens - we are setting up configurations from Azure KeyVault using Managed Service Identity
     //without specifiying any details of the Azure KeyVault itself (except the Url of the vault)
    .ConfigureAppConfiguration((context, config) =>
    {
       var builtConfig = config.Build();
       var keyVaultUrl = $"https://{builtConfig["KeyVaultName"]}.vault.azure.net";

       //this comes with .net core 2.1
       config.AddAzureKeyVault(keyVaultUrl);
                
       //if using 2.0, you should use this apporach
       //AzureServiceTokenProvider - this is the magic piece that makes it seamless to work with MSI
       //var azureServiceTokenProvider = new AzureServiceTokenProvider();
       //var keyVaultClient = new KeyVaultClient(
       //new KeyVaultClient.AuthenticationCallback(azureServiceTokenProvider.KeyVaultTokenCallback));
       //config.AddAzureKeyVault(keyVaultUrl, keyVaultClient, new DefaultKeyVaultSecretManager());
       })
       .UseStartup<Startup>()
       .Build();

if you look closely, all information we have about the Azure KeyVault in our web application is the name (or URL if you will) of the KeyVault (KeyVaultName) stored in our appsettings.json (or Application Settings in the published web app). Everything else is managed by MSI for us.

In ARM template, we enable MSI for the web application with this property:

"identity": {
     "type": "SystemAssigned"
   }

This enables this setting for the web application

So, as we did it Part I, let's deploy the template to provision new set of resources for us (You will find instructions in GitHub repository for this). Once resources are provisioned, this is how it looks for me in Azure Portal:

Four resources provisioned by ARM template in Azure Account

If i navigate to the application settings of my web application, this is what I see:

Notice, there is no connection string as we had earlier, instead there is an App Settings 'KeyVaultName' which contains the name of the vault. That's the only information this web application has. Let's deploy the code for the web application (like earlier, right-click and publish). After successfull deployment, this is what you should see:

Web Application without Secrets after successful deployment

This looks exactly same as previous web application since we haven't changed the logic. The only thing that was changed in this web app was to read connection string from KeyVault.

Now let's uncover some more details. If you navigate to the Azure KeyVault resource and then head over to Secrets, you will see this:

If you pay attention, you don't see anything, instead, it tells you that "You are unauthorized to view these contents." But the web application lists the URIs just fine! Let's go to Access Policies tab, you will see that there is one policy created for a web application:

Access Policy for Web Application

This was created by the ARM template after web application was provisioned. This is the section in ARM Template which defines this policy for the web application

"accessPolicies": [
          {
            "tenantId": "[reference(concat(resourceId('Microsoft.Web/sites', variables('webAppName')), '/providers/Microsoft.ManagedIdentity/Identities/default'), '2015-08-31-PREVIEW').tenantId]",
            "objectId": "[reference(concat(resourceId('Microsoft.Web/sites', variables('webAppName')), '/providers/Microsoft.ManagedIdentity/Identities/default'), '2015-08-31-PREVIEW').principalId]",
            "permissions": {
              "keys": [ "all" ],
              "secrets": [ "all" ]
            }
          }
        ],
        "tenantId": null,
        "tenantId": "[reference(concat(resourceId('Microsoft.Web/sites', variables('webAppName')), '/providers/Microsoft.ManagedIdentity/Identities/default'), '2015-08-31-PREVIEW').tenantId]",
        "sku": {
          "name": "Standard",
          "family": "A"
        }

Since at the moment, the only allowed access is for the web application you are not able to see any secrets when you go to Secrets tab. You can give access to your account if you want to see these secrets. Remember to click Save button after adding the policy, otherwise settings will not be saved

Adding Access Policy in Azure Key Vault

now, when I navigate the Secrets again, i'm able to see the Secrets.

Secrets visible after adding access policy for the logged in user in Azure Portal

Remember: If you want to use secrets from the KeyVault while development, you will have to assign access policy for your user in Azure KeyVault. Otherwise, you will get access denied exception.

Conclusion:

So as we saw, it's fairly straightforward to keep sensitive information in Azure KeyVault and configure our Web Application to read these secrets from the Vault. Moreover, Managed Service Identity feature further facilitates us to keep the information about KeyVault out of our web application. This can be applied to any scenarios - this is not necessarily bound to the web application only. You can keep connection strings, keys, certificates and all sorts of information you don't want to keep in your consumer application.

So, how do you handle your secrets? What is your opinion on this topic? I would love to hear more from you, please leave your comments.

Cheers

Identifying the owner of personal sites in SharePoint

Thu, 22 Jan 2015 00:00:00 GMT

Scenario/Problem:

Recently I was faced with a situation where I wanted to know the owner of the Personal Site in SharePoint. While it’s fairly straightforward to identify the personal site by the WebTemplate property in the corresponding SPWeb object but there was no property that could simply pin-point the owner of the site itself. If you take a look at the User Profile (from User Profile Service in Central Administration), there is a property called “Personal Site” (internal name is PersonalSpace) that tells us the relative url of the personal site for the respective profile. It is set automatically by SharePoint after the personal site is successfully provisioned. Wouldn’t it be nice in if we simply have a property or attribute on SPWeb object that tells us just this.

Solution:

However, it is not that difficult to achieve this. There could be many possible implementations but here I’m going to discuss one approach that I recently implemented.

So based on the information I could get out of SharePoint, I first set the Personal Site property to be indexed so that it can be crawled by the search service. Then, I created a managed property called PersonalSite and map it to PersonalSpace property. This gives me the possibility to query the search engine like “PersonalSite:/person/adamb”. If this is targeted to People scope, you will get the user profile whose personal site is set to /personal/adamb. And its safe to assume that the value would be unique.

Having set up the basics, now we can simply write a search query to give us the result we want.

Moving ahead, we can store this value in the property bag of the SPWeb object of the personal site. And whenever we need this property, we can first check if the value is present in the property bag, if we already have it, we don’t need to make a query again to the search, otherwise we can make a search query and store the value in the property bag for the next time.

Here is the code snippets for the above mentioned functionality. This code snippet doesn’t contain the creation of managed property but rest of the implementation.

With this fairly simple implementation, we can grab the owner of the personal site and fetch his user profile to perform other required operations.

Please feel free to share if you have any other alternative(s) to get this done.

P.S: You will require to run the Full Crawl after creating the managed property with the correct mapping. Before this, you will not get any result back from the search query.

Cheers !

Deleting Managed Metadata Fields via PowerShell

Thu, 08 Jan 2015 00:00:00 GMT

Scenario When you provision a Managed Metadata (Taxonomy) field in SharePoint, it creates a hidden field (of type Note) along with the main taxonomy field. What happens when you delete that field? You would expect the other one to be deleted as well, right? Not that easy :)

SharePoint for some reason doesn’t delete that hidden field. So when you provision the field again with the same name, it simply increments the number for the hidden field. Let’s say your field is called MyTax1. The hidden field will be named MyTax1_0. Now if MyTax1_0 already exists, SharePoint will create MyTax1_1 and so on. This is applicable when you working from the interface with OOB functionality.

Problem What if you are provisioning taxonomy fields through code or PowerShell where you provision certain fields, associate them with the content types etc. Moreover, during the development/testing phase, you might need to delete those fields and re-provision them number of times before your code is production ready. Well, if thats the case then most likely you will end with the following exception.

A duplicate field name dd93a2825ad24eef8e3e1cf4f2e96a40 was found

Reason? The hidden field is not deleted yet and your code is trying to create the field with the same name.

If you check it with SharePoint Manager 2013 you can see that this Guid is the internal name of the hidden taxonomy field

Solution In order to get around this issue, we need to make sure that hidden Note field is also removed. I wrote few lines of PowerShell script to delete those hidden fields as well. Here is the code snippet.

Hope it helps. Cheers.

Using HttpHandler to deal with cross-domain requests from AngularJS app for SharePoint on-premises

Sat, 16 Aug 2014 00:00:00 GMT

Scenario:

When you go to the newsfeed in your personal site (mysite newsfeed), you can write post and target it to different sites you are following or on your personal feed.

Recently, I had to built the same functionality for one of the projects. The ideas was to build the UI using SharePoint’s REST APIs and AngularJS. I am not using Office365 so i still have SharePoint on-prem.

Problem:

Its quite simple as you would expect to retrieve the list of sites a user is following. You simply call this REST endpoint “../_api/social.following/my/followed(types=4)” and you have the result, no brainer. But only when you dig into the results, you get to know that even though the list is correct but not every site returned has newsfeed enabled. As a result, you also get those sites that have no newsfeed and these are the sites you don’t want to display. So, how do you eliminate those?

The problem is that there is no information in the JSON response that REST endpoint returns. The only option you have is to actually check for the SiteFeed feature on that web. So you get the site, open that web and see if this feature in enabled. Now this is all fine when you are on server side but if you are implementing your functionality using client side, you will end up dealing with cross-domain issues and you will get exception as it is treated as a cross-domain request by the browser.

Now if you were dealing with Office365, this could have been resolved using Microsoft Azure’s ACS to authenticate your app with SharePoint and all would have worked just fine. But that’s not the case when you are building your client app using jQuery or Angular and using REST APIs in SharePoint on-prem.

Solution:

So, after struggling quite a bit to somehow deal with this, I ended up creating the HttpHandler in which I retrieved the list of sites the current user is following. Then I iterated through that collection to check for the Site Feed feature and return the collection of sites only with site feed feature enabled and continue to building my AngularJS app as I was before hitting this issue.

I am not putting up the entire code or even complete code for my app here. I am adding some snippets demonstrating the solution. Here is the markup in my view. Its simply a drop down that lists the sites I’m following just like the one you get in SharePoint by default.

<!-- Here is the markup in the view (html file) of my angularjs app. I am putting just the markup for rendering the dropdown with required results and eliminating rest of the code for demonstration purpose -->
<!-- here, vm.followedSites is the model that im setting in my controller and select element is bound to that model -->
 <div>
      <select ng-model="vm.selectedValue" ng-options="value.Name for value in vm.followedSites">
          <option value="">everyone</option>
      </select>
</div>

Here is my controller that calls the handler to retrieve that data and set the model that my view is bound to.

/* to simplify things, i am demonstrating the underlying method that is calling in the httphandler to get the desired result 
 this method is inside my controller 
 */
 
 function loadFollowedSites(currentUser) {
            //construct the URL for httphandler to get the sites with newsfeed activated
            var url = "/_layouts/15/myapp/Handlers/ClientHandler.ashx?operation=getFollowedSitesWithNewsFeed&username="+ currentUser;
            
            //make request to the url and return the result back
            return $http.get(url).then(function (response) {
                return response.data;
            });
        }

And finally, here is the code that retrieves the list of sites i’m following in my HttpHandler

/**
Here is the code snippet from my HttpHandler
 
**/  
  public void ProcessRequest(HttpContext context)
  {
      context.Response.ContentType = "application/json";
      context.Response.ContentEncoding = Encoding.UTF8;
      
      string operation = context.Request.QueryString["operation"];
 
      if (!String.IsNullOrEmpty(operation))
      {
        /* Code removed */ 
        if (operation == "getFollowedSitesWithNewsFeed")
        {
            string userName = (context.Request.QueryString["username"] != null) ? context.Request.QueryString["username"] : String.Empty;
            List<SPSocialActor> newsFeedFollowedSites = GetFollowedSitesWithNewsFeed(context, userName);
            context.Response.Write(JsonSerializer.Serialize(newsFeedFollowedSites));
        }
        
        /* code remoed */
      }
 
  }
  
  //Retrieves the list of all the sites that specific user is following that has SiteFeed web feature enabled
  private List<SPSocialActor> GetFollowedSitesWithNewsFeed(HttpContext cxt, string username)
  {
      List<SPSocialActor> followedSitesWithNewsFeed = new List<SPSocialActor>();            
      SPServiceContext context = SPServiceContext.GetContext(cxt);
      UserProfileManager profileManager = new UserProfileManager(context);
 
      if (profileManager.UserExists(username))
      {
          UserProfile userProfile = profileManager.GetUserProfile(username);
          SPSocialFollowingManager followingManager = new SPSocialFollowingManager(userProfile, context);
 
          SPSocialActor[] followedSites = followingManager.GetFollowed(SPSocialActorTypes.Sites);
 
          foreach (var actor in followedSites)
          {
              try
              {
                  using (SPSite site = new SPSite(actor.Uri.AbsoluteUri))
                  {
                      SPWeb rootWeb = site.RootWeb;
                      
                      //Check for SiteFeed feature (web)
                      Guid featureGuid = new Guid("15a572c6-e545-4d32-897a-bab6f5846e18");
                      if (rootWeb.Features[featureGuid] != null)
                      {
                          followedSitesWithNewsFeed.Add(actor);
                      }
                  }
              }
              catch 
              { 
                //Handle exception
              }
          }
      }
      else
      {
        //Log - user doesn't exist
      }            
      
      
      return followedSitesWithNewsFeed;
  }

Please let me know if there’s something I have missed or if there is any better way of doing this. I will really appreciate it.

Thanks Cheers

The specified program requires a new version of Windows. (Exception from HRESULT:0x8007047E)

Sat, 23 Nov 2013 00:00:00 GMT

Scenario: I have a custom list definition based on which I want to create some lists and associate OOB approval workflow with them for content approval.

In the schema.xml file for my custom list definition (based documents library i.e., basetype=1), I set ModeratedList=“True” attribute among others (see in the code snippet below)

<List xmlns:ows="Microsoft SharePoint" Title="TestList" Direction="$Resources:Direction;" Url="TestList" BaseType="1"
xmlns="http://schemas.microsoft.com/sharepoint/" EnableContentTypes="TRUE" EnableMinorVersions="TRUE" VersioningEnabled="TRUE" DraftVersionVisibility="2" ModeratedList="TRUE”>

Now, I added a ListAdded event receiver that listens to my custom list definition,so whenever any list is created based on my custom list definition, the OOB approval workflow should be associated with the list.

Here is the code snippet that associates the OOB approval workflow with the list


///
/// Associate OOB Approval workflow with the list
///
///
public static void AssociateApprovalWorklowWithList(SPList list)
{
  //variables
  Guid listId = list.ID;
  Guid webId = list.ParentWeb.ID;
  Guid siteId = list.ParentWeb.Site.ID;
  SPSite site = null;
  SPWeb web = null;

  try
  {
    site = new SPSite(siteId);
    web = site.OpenWeb(webId);
    bool allowUnsafeCurrent = web.AllowUnsafeUpdates;
    web.AllowUnsafeUpdates = true;
    list = web.Lists[listId];

    //Get Approval Workflow Template Base Id
    Guid workflowBaseId = new Guid("8ad4d8f0-93a7-4941-9657-cf3706f00"+ web.Language.ToString("X"));

    //If workflow is already associated, don't re-associate
    if (list.WorkflowAssociations.GetAssociationByBaseID(workflowBaseId) != null)
       return;

    //check if workflows feature is activated, if not activate the feature
    SPFeature workflowsFeature = web.Site.Features[WorkflowId];

    if (workflowsFeature == null)
      web.Site.Features.Add(WorkflowId);

    // Get workflow history and task list
    SPList historyList = EnsureHistoryList(web);
    SPList taskList = EnsureTasksList(web);
    string workflowAssociationName = string.Format("{0} - Approval", list.Title);
    SPWorkflowTemplate workflowTemplate = web.WorkflowTemplates.GetTemplateByBaseID(workflowBaseId);
    if (workflowTemplate == null)
    {
     //Log exception
     return;
    }

    workflowTemplate.AllowManual = false;

    // Create workflow association
    SPWorkflowAssociation workflowAssociation = SPWorkflowAssociation.CreateListAssociation(workflowTemplate,
    workflowAssociationName, taskList, historyList);

    var associationDataXml = XElement.Parse(workflowAssociation.AssociationData);
    // Add workflow association to my list
    list.WorkflowAssociations.Add(workflowAssociation);

    //Set workflow association data
    workflowAssociation.AssociationData = Add_Association_Data(web, associationDataXml);

    // Enable workflow
    if (!workflowAssociation.Enabled)
        workflowAssociation.Enabled = true;

    if (list.DraftVersionVisibility != DraftVisibilityType.Approver)
       list.DraftVersionVisibility = DraftVisibilityType.Approver;

    list.WorkflowAssociations.Update(workflowAssociation);
    list.DefaultContentApprovalWorkflowId = workflowAssociation.Id;
    list.Update();
    web.AllowUnsafeUpdates = allowUnsafeCurrent;
  }
  catch (Exception ex)
  {
    //Log Exception
  }
  finally
  {
    //Dispose the objects
    if (web != null)
       web.Dispose();
    if (site != null)
       site.Dispose();
   }
}

For the explanation of retrieving the workflow template id (line#23), please refer to my previous post here Associating OOB Approval workflow with custom list for different Locale (LCIDs)

Problem: Once I have deployed the solution and everything is hooked up, I create the list based on my custom list definition and the process ends with the following exception:

But when I go to the site, the list is created and workflow is associated properly.

Solution: After narrowing down the problem, I removed the ModeratedList attribute from List element in schema.xml file for custom list definition (shown in the first code snippet), enabled the content approval for the list in the code and redeployed the solution and everything worked !

Here is the code for enabling content approval through code (add this in the function defined above after setting DraftVersionVisibility property)


  if(!list.EnableModeration)
    list.EnableModeration = true;

Cheers

Associating OOB Approval workflow with custom list for different Locale (LCIDs)

Sun, 20 Oct 2013 00:00:00 GMT

Scenario: I have a custom web template that is used to provision the site. Among other things (branding etc), a custom documents library is provisioned whenever a new site is created (also based on my custom documents library template). Besides creating the custom library, I need to associate the OOB Approval workflow with the library that will be used as a publishing workflow. So if any user publishes the major version of the document it goes through the approval process.

Problem/Limitation: Associating the workflow with any list or library is fairly straight forward. Here are the steps:

Fetch the workflow template from parent web where list exist
Get the target list
Create the workflow association
Check if workflow is already associated with the list
If workflow is not associated, add the workflow association to the list

Here is the code snippet for performing this task.

private void AssociateApprovalWorklowWithList(SPList list)
{
SPWeb web = list.ParentWeb;
SPSite site = web.Site;
try
{
bool allowUnsafeCurrent = web.AllowUnsafeUpdates;
web.AllowUnsafeUpdates = true;
//Associate the approval workflow with libraries
//based on Custom Documents template only
if (list != null && list.BaseTemplate.ToString() == "10050")
{
//Get Approval Workflow Template Base Id
Guid workflowBaseId = new Guid("8ad4d8f0-93a7-4941-9657-cf3706f00409");
//If workflow is already associated, don't re-associate
if (list.WorkflowAssociations.GetAssociationByBaseID(workflowBaseId) != null)
return;
// Get workflow history and task list
SPList historyList = EnsureHistoryList(web);
SPList taskList = EnsureTasksList(web);
//set the name of workflow association
string workflowAssociationName = string.Format("{0} - Approval", list.Title);
//Get workflow template by Base Id
SPWorkflowTemplate workflowTemplate = web.WorkflowTemplates.GetTemplateByBaseID(workflowBaseId);
if (workflowTemplate == null)
{
//Log - no template found and return
return;
}
// Create workflow association
SPWorkflowAssociation workflowAssociation =
SPWorkflowAssociation.CreateListAssociation(workflowTemplate, workflowAssociationName,
taskList, historyList);
// Add workflow association to custom list
list.WorkflowAssociations.Add(workflowAssociation);
workflowAssociation.AssociationData = String.Empty;
//.. details omitted
list.WorkflowAssociations.Update(workflowAssociation);
list.DefaultContentApprovalWorkflowId = workflowAssociation.Id;
list.Update();
web.AllowUnsafeUpdates = allowUnsafeCurrent;
}
}
catch (Exception ex)
{
//Log exception
}
finally
{
//Dispose objects
}
}

The above code snippet will work just fine. But there is a little problem.

What if the site is created on language other than English? One case ask, why would there be any problem as we are fetching the workflow template by Base Id. Well, it turned out that the Base Id is not the same across different languages, it varies. So it failed if site has different locale.

There is also another method GetWorkflowTemplateByName but it didn't work out for me as well, even though I set the right culture.. So that means, we can not use both the methods.

Solution: Trying both the methods to get the template didn't work and I didn't get anything by googling it either. So the only way out of this problem was the following:

Looking at the BaseIds of different sites, a pattern emerged. Let's examine the BaseIds from English and Swedish sites, as we saw previously

8ad4d8f0-93a7-4941-9657-cf3706f00409 (English, LCID: 1033)
8ad4d8f0-93a7-4941-9657-cf3706f0041D (Swedish, LCID: 1053)

If you look closely, almost entire Id is same except last three digits Let's take some more examples:

8ad4d8f0-93a7-4941-9657-cf3706f00415 (Polish, LCID: 1045)
8ad4d8f0-93a7-4941-9657-cf3706f00406 (Danish, LCID: 1030)

as you can see, only the last three digits vary from language to language.

Now take the LCID and convert it to hexadecimal representation (thanks to my colleague Matthias Einig for breaking this code ;) )

Lets say, for:

LCID = 1033 => hex = 409

LCID = 1053 => hex = 41d

LCID = 1045 => hex = 415

So, it became clear that the last three digits that differ are basically the hexadecimal representation of LCID. Having identified this pattern, I simply converted the LCID to hex value and there we have it, the BaseId for approval workflow template for current language

So, if we change the line #15 in our code snippet above to the following:

Guid workflowBaseId = new Guid("8ad4d8f0-93a7-4941-9657-cf3706f00"+ web.Language.ToString("X"));

With this change, now no matter which locale the site has, it will construct the right workflow template id and will fetch the workflow template for association.

Hope it helps some folks out there :)

Updates are currently disabled in GET requests exception while provisioning multiple site collections

Sun, 02 Jun 2013 00:00:00 GMT

Scenario:

Recently i was working with creating site provisioning in SharePoint 2010. For that I have a list where user submits the request for the site that needs to be created (along with some required metadata). Then I have scheduled the timer job that picks up items from that list (based on the status), iterates through the items and provisions the site.

Here is the method in Timer Job that iterates through the items and creates site collections.

public static void ProvisionSite(String siteCollectionUrl)
{
   using (SPSite siteCollection = new SPSite(siteCollectionUrl))
   {
      SPList requestList = siteCollection.RootWeb.Lists.TryGetList("SiteRequests");
      if (requestList != null)
      {
        var query = new SPQuery();

        //Fetch the items that have Status=Requested
        query.Query = String.Format("{0}", "Requested");

        SPListItemCollection items = list.GetItems(query);

        if (items != null && items.Count > 0)
            ProcessRequest(items);
      }
   }
}

//Generates random 7-digit number
public static String GenerateSiteNumber()
{
  Random random = new Random();
  String siteNumber = random.Next(0000000, 9999999).ToString();
  return siteNumber;
}

public static void ProcessRequest(SPListItemCollection items)
{
   try
   {
      foreach (SPListItem item in items)
      {
        try
        {
           SPSite site = null;
           SPSecurity.RunWithElevatedPrivileges(delegate
           {
               SPWebApplication webApp = item.Web.Site.WebApplication;
               SPSiteCollection siteCollections = webApp.Sites;

              //Generate random number to use as site collection url
              String siteNumber = GenerateSiteNumber();
              String relativeUrl = "/sites/" + siteNumber;

              //Fail if site collection already exists
              if (SPSite.Exists(new Uri("http://sp2010demo/" + relativeUrl)))
                   throw new Exception("Site already exists");

               using (SPSite newSite = siteCollections.Add(relativeUrl, "My Site" + siteNumber, "Test Site collection", 1033, null, "dc\spadmin", "",       ""))
               {
                  site = newSite;
                  //.. code removed for clarity .....
               }
           });
        }
      }
   }
   catch
   { }
}

Problem:

As provisioning is done by the timer job Context is null (i.e., HttpContext.Current will be null). In my solution, I am simply iterating through the list items and creating the site collections. First request always works fine, site is created successfully but I am getting an exception as soon as the site for the next item is being provisioned. The exception that I get is the following:

"Updates are currently disabled in GET requests. To enable updates in GET requests, change the property AllowUnsafeUpdates on SPWeb."

Now apparently, this exception is thrown when you perform any GET operation on Web and you haven't set AllowUnsafeUpdates to "True" !! But I'm just creating a site collection, so I don't have any SPWeb object where I should set this property ??

After investigating a little further (and with the help of ILSpy) it turned out that after first site collection is created, HttpContext.Current is set to the context of this newly created site collection and for the next item, Context is already set to the previous site collection and exception is thrown at SiteCollection.Add() method. The Add() method basically checks if HttpContext is already set, if yes then it uses the context and in my case this is wrong!

Solution

The solution is quite easy to fix.

After the completion of first iteration, set the HttpContext to null so that during the next request there is no context set.

Here is the line of code (finally block) that I added to my previous method.


public static void ProcessRequest(SPListItemCollection items)
{
 try
 {
   foreach (SPListItem item in items)
   {
     try
     {
         //... code removed for clarity --
     }
     finally
     {
        //Reset the Context
        System.Web.HttpContext.Current = null;
     }
   }
 }
 catch
 { }
}

Hope it helps.

Incorrect values for User Fields while copying list items from one site collection to another

Sat, 25 May 2013 00:00:00 GMT

Scenario

Recently, I was writting a piece of code to copy list item from a list in one site collection to the list in another site collection. I did the shallow copy (copying field by field). Both the lists were created using same List definition so they had same fields and content types etc. The item was successfully copied but then I noticed that the values of user fields were not correct.

Here is the pice of code that i used for copying the item


Public static void Main(string[] args)
{
    using (SPSite sourceSite = new SPSite("http://sp2010demo/sites/source"))
    {
      using (SPWeb web = sourceSite .OpenWeb())
      {
          SPList sourceList= web.Lists.TryGetList("Source List");
          //fetch first item - for demonstration purpose
          SPListItem sourceItem = sourceList.Items[0];
          //instantiate destination site collection
          using (SPSite destinationSite = new SPSite("http://sp2010demo/sites/destination/"))
          {
             var destinationList = destinationSite.RootWeb.Lists.TryGetList("DestinationList");
             if (destinationList != null)
             {
                //Add new item
                SPListItem destinationItem = destinationList.Items.Add();
                //copy item
                CopyListItem(sourceItem, destinationItem);
             }
          }
      }
   }
}

public static void CopyListItem(SPListItem sourceItem, SPListItem destinationItem)
{
        foreach (SPField field in sourceItem.Fields)
        { 
          //Filter out readonly and attachment fields
          if ((!field.ReadOnlyField) &amp;&amp; (field.InternalName != "Attachments"))
          {
                destinationItem[field.Title] = sourceItem[field.Title];
          }
        } 
        destinationItem.Update();
}

Apparently, it looked just fine and i used the same piece of code to copy the list items from one list to another within the same site collection and it worked just perfect as it should !

Problem

So, the problem was actually that when you move this item to a different site collection, users have to be present there in order for it to have the correct value.

If you look at the internal representation of the user field it is basically a look up value and the value is represented something like this: 1;#Sanjay Bhagia

Where first number is the user id in User Information List in the site collection that can be accessed (only if you are admin) by navigating to the following url in your site _catalogs/users/simple.aspx (e.g., http://sp2010demo/sites/source/_catalogs/users/simple.aspx).

So lets do some investigation.

I first open my source site collection i.e., http://sp2010demo/sites/source and navigate to my user information list by navigating to the following url http://sp2010demo/sites/source_catalogs/users/simple.aspx It shows all the users that have been added in my site

Now, if you put the cursor on the user link, you can see the entire url in the status bar of your browser. At the end of the url you can see the querystring ID that has value 1 in this example.

Now lets navigate to the same user list in destination site collection by (http://sp2010demo/sites/destination/_catalog/users/simple.aspx).

Note: In my environment, I already have my user added in both the site collections, that is why I'm able to see my user at both the places but it is possible that you don't see any user at both the places. User will be added automatically if you have logged in with that user on that site

By observing the url of the same user we can see that the user id for this user in destination site collection is 10.

So if I'm copying the list item that has user field and that field has my user added (i.e., Sanjay Bhagia) the representation of that user in my source site collection will be 1;#Sanjay Bhagia but the representation of the same user in my destination site collection will be 10;#Sanjay Bhagia !! Hence, the copied item will not display the proper user at the destination list item.

Solution It is quite easy to fix this issue, I basically fetched the value from user field and called SPWeb.EnsureUser() method to make sure the user is added in the User Information List in my destination site collection and updated the LookupId attribute of the SPUser object.

Here is the modified code:


public static void CopyListItem(SPListItem sourceItem, SPListItem destinationItem)
{
    foreach (SPField field in sourceItem.Fields)
    {
        //Filter out readonly and attachment fields
	if ((!field.ReadOnlyField) && (field.InternalName != "Attachments"))
        {
            //only for user type fields
            if (field.GetType() == typeof(SPFieldUser))
            {
               SPWeb destinationParentWeb = destinationItem.ParentList.ParentWeb;
               if (sourceItem[field.Title] != null)
               {
                 //for field can have multiple selection enabled otherwise you can user SPFieldUserValue class
                 SPFieldUserValueCollection users = new SPFieldUserValueCollection(sourceItem.ParentList.ParentWeb, sourceItem[field.Title].ToString());
                 foreach (SPFieldUserValue user in users)
                 {
                    //this LookupId is the number of user object in User Information List
                    user.LookupId = destinationParentWeb.EnsureUser(user.User.LoginName).ID;
                 }
                destinationItem[field.Title] = users;
               }
            }
            else
              destinationItem[field.Title] = sourceItem[field.Title];
                    }
        }
    destinationItem.Update();
}

So basically, I'm making sure that user object in destination site collection has the correct look up id, that's it!

Hope it helps.

Creating loosely coupled and generic process steps implementation

Thu, 02 May 2013 00:00:00 GMT

Recently, during one of the SharePoint implementations we came across the requirement to build the workflow for site provisioning engine that we are developing.

One of the requirements of the process is to rollback everything that we create/modify if there is any problem/exception during the process. So for example, if we created the site collection and then in the next step process failed at setting up custom permissions, the process should be roll backed to the initial stage.

Normally, this is no big deal to implement everything as a standard workflow and you write a piece of code for different activities of the workflow (or use standard workflow activities from Visual Studio toolbox) but then an idea came up why not write an implementation of the process steps independent of the workflow in a nice and loosely couple way that should take care of roll back in a much flexible and nicer and cleaner way !

So in this post, I will try to describe the solution that I came up with and also implementation details.

So here is the implementation details:

First, I created an interface that contains the method declarations for Execute and RollBack methods


public interface IProcessStep
{
   void Execute(ProcessStepProperties properties);

   void RollBack(ProcessStepProperties properties);
}

As you see in the code above, I'm using parameter of type ProcessStepProperties. This is basically a custom class that I use as a data transfer object (DTO) across different steps and methods. It's basically a collection (dictionary in my implementation) that contain key/value pairs. Here is the implementation:


public class ProcessStepProperties
{
   public Dictionary<String, Object> Keys = new Dictionary<String, Object>();

   public bool KeyExists(String key)
   {
     if (Keys.ContainsKey(key) && Keys[key] != null)
        return true;
     return false;
   }
}

Having interface ready, we write a class that defines one the process steps. This step is CreateSiteStep that creates the site collection. This class inherits from our interface IProcessStep and hence it provides the implementation of Execute and RollBack methods.


public class CreateSiteStep : IProcessStep
{
   //private method to check for preconditions for Execute method
   private bool PrequisitesPresent(ProcessStepProperties properties)
   {
      if (properties.KeyExists("SiteNumber") && properties.KeyExists("Title") && properties.KeyExists("TemplateName"))
         return true;
      return false;
   }

   public void Execute(ProcessStepProperties properties)
   {
      //Check if preconditions are met
      if (PrequisitesPresent(properties))
      {
         //Create site collection
         CreateSiteFromRequest(properties);

         //Update the property bag
         var spSite = properties.Keys["SPSite"] as SPSite;
         if (spSite != null)
         {
            properties.Keys["SiteUrl"] = spSite.Url;
         }
      }

      //Write checkpoint for rollback
      TransactionLogger.Log(this);
   }

   public void RollBack(ProcessStepProperties properties)
   {
        // Add Logging information  - ULS or event logger if appropriate
        SPSite site = properties.Keys["SPSite"] as SPSite;
        DeleteSite(site);
   }

Now there are couple of things that need to be discussed in the above code.

Firstly, as you see, I have created a private method to check pre-conditions before the Execute method is executed. This is totally upto the specific step to check if it requires any pre-conditions to be met or any values it requires before it proceeds.

Secondly, as i mentioned above I'm using ProcessProperties class (key/value collection) to transfer the state across methods and steps, I'm updating (adding) the value in the collection right after Site collection has been created. In this way, the next step (or rollback) step will have the updated information that might be useful for further execution of the process.

Thirdly and quite importantly there is a call to Log method of TransactionLogger class. TransactionLogger.Log(this);

This is very important. Again, TransactionLogger is my custom class that I'm using to log in the information for steps that have successfully been completed. This serves the purpose of check point for me that i use to roll back when anything fails. I'm using Stack as a data structure to push the steps (of type IProcessStep) whenever any step is successfully executed and if there is any failure during the execution, i simply pop the steps and invoke RollBack method until stack is empty - hence everything is cleaned up in case of failover.

Here is the implementation of TransactionLogger class.


public class TransactionLogger
{
  //to keep checkpoints
  private static readonly Stack TransactionStack = new Stack();

  //public property to check number of steps in the stack
  public static int LogCount
  {
      get { return TransactionStack.Count; }
  }

  private IProcessStep Pull()
  {
     return TransactionStack.Pop();
  }

  //Pushes the process step into the stack
  public static void Log(IProcessStep step)
  {
      TransactionStack.Push(step);
  }

  //Pops the step from stack and invoke rollback method for every step  until stack is empty
  public static void RollBack(ProcessStepProperties properties)
  {
      while (TransactionStack.Count > 0)
      {
          var step = TransactionStack.Pop();
          step.RollBack(properties);
      }
   }
}

So, basically the idea is that in case of any failure everything is rolled back and in the implementation above, TransactionLogger class doesn't need to know the details about any ProcessStep class how to roll back because it is upto every step to roll back the things it has created. That is why TransactionLogger class adds IProcessStep object into the stack and that is our interface! here is another class called SetSitePermissionGroupStep that basically adds custom permissions to the site collection that has been created in previous process step.


public class SetSitePermissionGroupStep : IProcessStep
{

   // .... code removed for clarity purpose ....

 public override void Execute(ProcessStepProperties properties)
 {
    if (PrequisitesPresent(properties))
    {
       AddAssociatedMemberGroups(properties, _key);
       TransactionLogger.Log(this);
    }
 }

 public override void RollBack(ProcessStepProperties properties)
 {
   //Roll back implementation
 }
}

So, as a recap, we defined an interface called IProcessStep that contains the declaration of Execute and RollBack methods.

Other classes (specific to process steps) inherit from IProcessStep and provide an implementation for Execute and RollBack methods. Execute method of every process step class also add check point after successful execution of the step using TransactionLogger.Log method.

Now having all this setup we need some sort of controller class that could use this implementation. This could be used from within the workflow as well but in my demonstration I'm using a class that implements the Queue and in that queue we can insert separate steps and then Process the queue in the end. Here is the implementation of my controller class.


public class TestProcessSteps
{
    private Queue _queue = new Queue();
    private ProcessStepProperties _properties = new ProcessStepProperties();

    public void Initialize(ProcessStepProperties properties)
    {
       _properties = properties;
    }
    public void Enqueue(IProcessStep step)
    {
       this._queue.Enqueue(step);
    }

    public void ProcessQueue()
    {
      try
      {
         while (_queue.Count > 0)
         {
            //pick item from the queue and invoke execute method
            var step = _queue.Dequeue();
            step.Execute(_properties);
         }
       }
       catch (Exception ex)
       {
          //roll back
          TransactionLogger.RollBack(_properties);
       }
    }
 }

If you notice above, if anything fails (i.e, exception occurs), RollBack method is invoked from TransactionLogger class (that basically pops up items from the stack and invoke rollback method on respective step).

Finally, here is how we test our implementation.


//Instantiate TestProcessStep class
TestProcessSteps testclient = new TestProcessSteps();

//setup properties
properties = InitializeProcessProperties(item);

//Invoke Initialize method on TestProcessSteps class (that basically initializes ProcessProperties)
testclient.Initialize(properties);

//Insert steps into Queue
testclient.Enqueue(new CreateSiteStep());
testclient.Enqueue(new SetSitePermissionGroupStep("MemberGroup"));
testclient.Enqueue(new SetSitePermissionGroupStep("OwnerGroup"));

//Process the Queue (that basically takes items from queue and invoke Execute methods on respective steps)
testclient.ProcessQueue();

Note that during the enqueue, I'm adding SetSitePermissionGroupStep step two times with different parameters - in this way you can actually define atomic steps also using the same class.

Hope it gives the general idea about my thought behind the implementation. Don't hesitate to provide any feedback or issues if you find any in my solution.

Thanks for taking time to read this long post :)

Consuming Content Type Hub in custom web templates

Thu, 21 Mar 2013 00:00:00 GMT

Setting up content type hub in SharePoint 2010 is quite straightforward process. There are simple steps to follow and if followed in correct order you can get it working in no time. Content Type Hub relies on managed metadata service. Unless all web applications are sharing the same instance of managed metadata service application, they should be able to consume content types from the content type hub (a site collection marked as a hub and other web application are marked to consume from this location).

However, if you are doing the customization and you have some custom Web Templates that you use to create sites and subsites. And in those sites, you want to consume content types from content type hub, you might get a little bump in the road during the setup.

As I mentioned, Content Type Hub relies on Managed Matadata service, one feature TaxonomyFieldAdded (a hidden) feature is enabled on every site collection by default. This feature is not activated by default when you create web templates (please refer to this blog for further explanation of this: Web Templates and Content Type Publishing). So if you navigate to the Site Settings for any site that is created from custom web template, you will not see any link for Term store management or Content Type Publishing

So for custom web templates, you are required to activate this feature. Here is the PowerShell code for enabling this feature on a specific site collection.

$snapin = Get-PSSnapin | Where-Object {$_.Name -eq 'Microsoft.SharePoint.Powershell'}

if ($snapin -eq $null)
{
Write-Host "Loading SharePoint Powershell Snapin"
Add-PSSnapin "Microsoft.SharePoint.Powershell" -EA SilentlyContinue
}

#Enable <strong>TaxonomyFieldAdded </strong>feature on site collection
Enable-SPFeature –Identity 73EF14B1-13A9-416b-A9B5-ECECA2B0604C –url "http://portal.local.domain/sites/subsite1/"

Once feature is activated, you will see the links in Site Settings appear !

Now, you should be able to consume content types from content type hub as expected. You can activate this feature directly from your web template, add an entry for this feature in onet.xml file for your web template, in this way you don't have to enable this feature manually.

</pre>
<SiteFeatures>
<!-- OOTB: Taxonomy -->
<Feature ID="73EF14B1-13A9-416b-A9B5-ECECA2B0604C" />
</SiteFeatures>
<pre>

Paged Custom Search Query using KeywordQuery

Thu, 14 Mar 2013 00:00:00 GMT

SharePoint Server Object Model provides two classes to perform custom search queries namely FullTextSqlQuery and KeywordQuery. Both of these classes inherit from a generic Query class (in Microsoft.SharePoint.Search namespace).

You can perform search operations using both of these classes depending on your needs as FullTextSqlQuery provides more sophisticated queries using Sql syntax whereas KeywordQuery class provides simpler syntax to do the search.

However, the scenario I'm explaining in this post is relevant to both of these classes but i will use KeywordQuery class to demonstrate.

Scenario:

I want to write a custom people search web part that takes in the keywords entered by user and performs the search.

I'm using powershell to demonstrate here but this can be done in C# as well in similar fashion.

So, here is the function that performs the basic search


Function GetSearchResults([string]$siteCollectionUrl, [string]$keyword, [int]$sIndex, [int]$rLimit)
{
	#load the site and set the keyword query object
	$searchSite = Get-SPSite $siteCollectionUrl

	$keywordQuery = New-Object Microsoft.Office.Server.Search.Query.KeywordQuery $searchSite

	$keywordQuery.ResultTypes = [Microsoft.Office.Server.Search.Query.ResultType]::RelevantResults

  	#Write-Host "Start Index " $sIndex
	#Write-Host "Row Limit " $rLimit
	#Write-Host "Keyword " $keyword

	$keywordQuery.StartRow = $sIndex
	$keywordQuery.RowLimit = $rLimit
	$keywordQuery.HiddenConstraints = "scope:People"

	#Pass the query text - SharePoint
	$keywordQuery.QueryText = $keyword

	#query execution
	$results = $keywordQuery.Execute()
	return $results
}

KeywordQuery class has two properties (RowLimit and StartRow) that are of our interest here. RowLimit specifies the number of items that you want the search to return when you execute the query and StartRow gets or sets the first row of information from the search results. So basically with both of these properties you get the paged search result instead of getting entire data in one chunk !

Now, one of the obvious reasons to do this performance. But there is one more reason as well. RowLimit (default value is 50 if you don't specify it yourself), even though can accept maximum 32-bit integer value theoretically, is set to some hardcoded limit of 10000. It means that if you set RowLimit to anything greater that 10000 you will get an exception when you execute the query.

The exception is something like this:

"Exception from HRESULT: 0x80040E01"

Here is the StackTrace from ULS log:

Log Query: More Information: Row could not be inserted into the rowset without exceeding provider's maximum number of active rows.

SearchServiceApplication::Execute--Exception: System.Runtime.InteropServices.COMException (0x80040E01): Exception from HRESULT: 0x80040E01
at System.Runtime.InteropServices.Marshal.ThrowExceptionForHRInternal(Int32 errorCode, IntPtr errorInfo)
at Microsoft.Office.Server.Search.Query.KeywordQueryInternal.Execute()
at Microsoft.Office.Server.Search.Query.QueryInternal.Execute(QueryProperties properties)
at Microsoft.Office.Server.Search.Administration.SearchServiceApplication.Execute(QueryProperties properties)

SearchServiceApplicationProxy::Execute--Error occured: System.ServiceModel.FaultException`1[System.ServiceModel.ExceptionDetail]:
Exception from HRESULT: 0x80040E01 (Fault Detail is equal to An ExceptionDetail, likely created by IncludeExceptionDetailInFaults=true,
whose value is: System.Runtime.InteropServices.COMException: Exception from HRESULT: 0x80040E01
at System.Runtime.InteropServices.Marshal.ThrowExceptionForHRInternal(Int32 errorCode, IntPtr errorInfo)
at Microsoft.Office.Server.Search.Query.KeywordQueryInternal.Execute()
at Microsoft.Office.Server.Search.Query.QueryInternal.Execute(QueryProperties properties)
at Microsoft.Office.Server.Search.Administration.SearchServiceApplication.Execute(QueryProperties properties)
at SyncInvokeExecute(Object , Object[] , Object[] )
at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)
at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage4(MessageRpc& rpc) ...).

You can determine this limit by this method in your environment.

# Get the maximum results returned value set for KeywordQuery and FullTextQuery in current environment
Function GetMaxResultsReturned()
{
 # Get reference to Search Service Application
 $ssa = Get-SPEnterpriseSearchServiceApplication
 return $ssa.GetSetting('Config:qp_MaxResultsReturned')
}

This value is configured at Search Service Application level. Now this could be different for your environment but default is 10000. So if you set RowLimit to anything greater than this value ('Config:qp_MaxResultsReturned') you will get an exception. You can modify this value if you want using this function

# Sets the maximum results returned value set for KeywordQuery and FullTextQuery in current environment
Function SetMaxResultsReturned($newLimit)
{
	$ssa = Get-SPEnterpriseSearchServiceApplication
	$ssa.UpdateSetting('Config:qp_MaxResultsReturned', $newLimit)
	$ssa.Update()
}

So, in order to avoid this exception, you should query to retrieve results in chunks and then merge them. In this way you don't loose on performance and also you will get your results back without facing this exception.

Following, I'm calling the GetSearchResults function (defined above) to perform people search query by retrieving results in pages and merging them in a DataTable.


$startIndex = 0
$rowLimit = 50
$siteCollectionUrl = "<Url of the site collection>"
$searchKey = "s*"

$resultCollection = GetSearchResults $siteCollectionUrl $searchKey $startIndex $rowLimit
$hits = $resultCollection.TotalRows
Write-Host Total hits: $hits
$resultsDataTable = $resultCollection.Table

#Iterate through the rest of the pages to fetch items
while($resultCollection.TotalRows -gt $resultsDataTable.Rows.Count)
{
	$sIndex = $resultsDataTable.Rows.Count
	$resultCollection = GetSearchResults $searchKey $startIndex $rowLimit
        $resultsDataTable.Load($resultCollection, [System.Data.LoadOption]::PreserveChanges)
}

# display the results in table format for better view
$resultsDataTable | Format-Table -AutoSize -Property title , url

Hope it helps!

Update Personal Site Property in User Profile

Wed, 06 Mar 2013 00:00:00 GMT

User Profile has one property called PersonalScope that stores the URL of the personal site of user (if mysite is enabled for users). When user creates personal site, this property is set. Recently we had some problem in our environment with user profile synchronization that removed some of the information for user profiles and Personal Site was one of them. The impact of this was that when affected users navigate to their personal site ("My Content") they get the message that personal site hasn't been created yet. On checking the site collections for my site web application in Central Administration, the site collections for users were present. So it was just the linkage that was broken.

It is as simple as updating the User Profile's Personal Site property with proper relative URL of the site collection, but if many users are affected then PowerShell is the way to go!

Here is the scrip that i wrote to fix this linkage. I basically, went through all the site collections (in My Site Web Application) and fetched the user profile for every site and checked if Personal Site is null, simply update it with relative URL.

Add-PSSnapin "Microsoft.SharePoint.PowerShell"

#This function resets the link to personal site for users who have personal site created but the link has been removed from user profile
Function UpdatePersonalSiteLinks($webApplicationUrl)
{
   #Instantiate Web Application Object
   $webApplication = Get-SPWebApplication -Identity $webApplicationUrl
   
   #Create Service Context for User Profile Manager
   $context = Get-SPServiceContext $webApplication.Sites[0]

   #Get User Profile Manager instance
   $profileManager = new-object Microsoft.Office.Server.UserProfiles.UserProfileManager($context)
   [Microsoft.SharePoint.SPSecurity]::RunWithElevatedPrivileges(
   {
     #Iterate through site collection in My Site Host web application
     foreach($siteCol in $webApplication.Sites)
     {
          Write-Host "Site: " $siteCol.Url

          #Get the user profile for owner of the personal site
          $uProfile = $profileManager.GetUserProfile($siteCol.Owner.LoginName)

          #Check if user profile has reference to personal site
          if($uProfile.PersonalSite -eq $null)
          {
             #If link has been removed, update the user profile
             Write-Host -ForegroundColor Red "Personal Site is not linked"
             Write-Host -ForegroundColor Green "Updating the link" $siteCol.RootWeb.ServerRelativeUrl

             #update the property ("PersonalSpace")
             $uProfile["PersonalSpace"].Value = $siteCol.RootWeb.ServerRelativeUrl
             $uProfile.Commit()
          }
     }
   })
}

UpdatePersonalSiteLinks "http://mysite.local.domain/"

It is important to run this script under right context, otherwise it might fail (to fetch the personal sites or updating the user profiles). That is why i have used RunWithElevatedPrivileges, however this might not be sufficient in your environment. So be sure you execute this script with right user credentials.

Update list items to trigger event handler through PowerShell

Mon, 25 Feb 2013 00:00:00 GMT

Scenario: A SharePoint list has different content types associated with it and every content type has a field of type taxonomy (connected to managed metadata termstore). And that list has an event handler attached to it that is triggered on item add and item update. In that event handler, you push the change to user profile property (for respective user profile) that is also of type managed metadata. So basically when you add or update an item in the list, that value is written back in a user profile property.

Now, under normal circumstances everything should work fine as event handler will kick in and write back the values to user property. However, if you have to delete the user profile or due to different reasons, the values in user profile is wiped off etc, you need to ensure that the data already inserted in the list is also present in the user profile.

Solution:

For such scenarios, what we need to do is trigger the event handler on every list item so that the value is written back to user profile property. PowerShell comes to our rescue in such situation!

Here is the powerShell script for that


Function UpdateListItems($webAppUrl)
 {
  [Microsoft.SharePoint.SPSecurity]::RunWithElevatedPrivileges(
  {
     #Get Site Collections for respective web applications
     $sitecolls = Get-SPSite -WebApplication $webAppUrl

     Write-Host "Total number of site collections: " $sitecolls.Count

     #Iterate through every site collection to find the list at the root web
     foreach($siteCol in $sitecolls)
     {
        $site = Get-SPSite $siteCol.Url

        Write-Host "Site Collection: " $site.Url
        $rootSite = $site.RootWeb

        Write-Host $rootSite.Name

        #Retrieve the list
        $list = $rootSite.Lists["ListName"]

        if($list -ne $null)
        {

         # Fetch list items only for required content type (If this needs to execute for every content type,
         # just remove the filter from following statement, just write $listItems = $list.Items)
         $listItems = $list.Items | ?{$_.ContentType.Name -eq "ContentTypeName"}

         foreach($item in $listItems)
         {
           Write-Host $item["Field's Dislay Name"]

           #Update the item
           $item.Update()
         }
       }
     }
  })
 }

Now once the function is defined, call it for execution.


UpdateListItems "http://mysite.mydomain.com"

One thing to notice in the script above is that we are using RunWithElevatedPrivileges. My Colleague Matthias Einig has written an amazing post on his blog regarding this, do check it out.

Hope it helps.

Remove webpart from a page on Personal site using delegate control

Sat, 12 Jan 2013 00:00:00 GMT

Customizing Personal Sites (site collections for users in my site) could be little challenging at times. One reason being that personal site is provisioned automatically by SharePoint and there could be more than one point from which user can provision the site and there is no support to create your own web template or site definition, as we can do with every other type of site (as per my knowledge). Creating your own web template provides great flexibility and easiness to provision sites having your own settings and all but lacking this feature for personal sites provide us more challenges, even for small customizations at times. In a recent project, one of the customer's wish was to remove "Recent Blog Posts" web part from default page when user provisions the personal site.

Solution: After trying out different approaches the final solution turned out to create a delegate control that removes the web part from the page when site is provisioned and having this delegate control added on the site collection through feature stappling.

So, here is what I did to fulfill this requirement.

Create a delegate control

Here is the element file for delegate control


<!--?xml version="1.0" encoding="utf-8"?-->

 <Control
 ControlAssembly="MyAssembly, Version=1.0.0.0, Culture=neutral, PublicKeyToken=b848d3ec396f6039"
 ControlClass="MyAssembly.RemoveWebPartsFromPersonalSiteDefaultPage" Id="AdditionalPageHead" Sequence="50"

<li>Fetch the desired webpart and remove it</li>

and in the code behind file for this delegate control add the functionality to fetch and remove the desired webpart (I've added this logic in Render method)



//... code removed

//Fetch Web Part title from SPCore resoruce file for current web language
 //It is important to fetch the localed based on current site's language not based on Browser locale or regional settings!
 var localizedRecentPostWebPartName = SPUtility.GetLocalizedString("$Resources:spscore,MySiteOnet_WebPart_Blog", "spscore", SPContext.Current.Web.Language);

//remove the webpart
 RemoveWebPartsFromWebPartPage(SPContext.Current.Web, "default.aspx", "MiddleRightZone", localizedRecentPostWebPartName);

 //add page redirectino logic otherwise it will be an infinite loop
 // ... code removed

Here is the definition of RemoveWebPartsFromWebPartPage method:



//... code removed

using (SP.SPLimitedWebPartManager webpartManager = web.GetLimitedWebPartManager(webPartPage.Url, Web.PersonalizationScope.Shared))
{
  var collection = webpartManager.WebParts;
  List webParts = new List();

  foreach (WebPart webPart in collection)
  {
    if (fromZoneId == webpartManager.GetZoneID(webPart))
    {
      if (webPart.DisplayTitle == webPartTitle)
      {
          webParts.Add(webPart);
      }
    }
  }

 // delete the web parts
 foreach (WebPart webPart in webParts)
 {
   webpartManager.DeleteWebPart(webPart);
 }
}

That's it! that is the functionality that we need to remove the webpart. We created a delegate control that contains the code to remove the web part from desired page! Now comes the part where we need to execute this delegate. You can use different approaches depending on your requirement. For this demo, we create a simple site scoped feature and add this delegate as an item in that feature.

Build your solution, create the package and deploy it. Activate the feature. Having this feature activated, as soon as user navigates to the default page, our delegate control will kick in and looks for the desired web part and removes it!

Other approaches could be creating feature stappling mechanism that attaches our delegate control with desire web templates so that this control is attached to every site that is created.

Hope it helps!

Adding local variables on a page as a user control

Thu, 20 Dec 2012 00:00:00 GMT

Quite often you need to use both server side and client side approach to build SharePoint solution as both offer different level of flexibility and functionality. One of the benefits of using mixed approach is that you can utilize the strength of both approaches and solve the problems quite quickly and efficiently.

In this post I'm going to present a part of the solution where we had to cache one control during one of the recent projects.

It is a very common practice that we write separate .js file for our client side scripting and than we can add these file either on a page or on a master page that we can use later on in our controls underneath. This works totally fine but the only thing we need to be careful about external files is that we need to wait for them to load before we can use anything inside these files. That is also fine but in some situations but there might be a little delay until file is loaded and that delay might be of quite a significance.

To overcome this issue, we (Me and Magnus Hansson, who is the mastermind of this solution) recently came up with an approach where we actually created a user control and in that user control, we added some script to set some local variables that we needed to use in on our page.

Here is the markup of the user control (ascx control):

Markup:


<script type="text/javascript">
var Local = window.Local || {};
   Local.Variables = function () {
     return {
        currentRootWebUrl : '<asp:Literal runat="server" id="litRootWebUrl" />',
        currentWebUrl : '<asp:Literal runat="server" id="litCurrentWebUrl" />',
        currentUser : '<asp:Literal runat="server" id="litCurrentUser" />',
        currentLCID : '<asp:Literal runat="server" id="litCurrentLCID" />'
       }
} ();
</script>

and in the CodeBehind(.cs) we set these asp controls embedded in script above:


protected override void CreateChildControls()
{
      //Get current user's Web object and Root Web Object
      SPWeb currentWeb = SPContext.Current.Web;
      SPWeb currentRootWeb = currentWeb.Site.RootWeb;

      //Get Locale and user's login name
      litCurrentLCID.Text = SPContext.Current.Web.CurrencyLocaleID.ToString();
      litCurrentUser.Text = SPContext.Current.Web.CurrentUser.LoginName;
      litCurrentWebUrl.Text = currentWeb.Url;
      litRootWebUrl.Text = currentRootWeb.Url;

      base.CreateChildControls();
}

So what we have achieved so far is basically set of variables that are set from code behind in control. When executed, the output would be a JavaScript object named "Local" that has a public property called "Variables" that contains some more properties.

Now in order to see this, we need to add this control either on application page or master page (or any where you find it appropriate in your case).

We added in on Master Page as we needed to used this on a control inside master page. In order to add it on a master page, you can directly Register your user control on master page and then added reference to the control in header section or even create a delegate control to attach it with master page through feature.

Once, control is added, navigate to your site and see the browser output. You should find this section rendered in the view source:


var Local = window.Local || {};

Local.Variables = function () {

return {
    currentRootWebUrl : 'http://demo.local.com',
    currentWebUrl : 'http://demo.local.com/site/hr',
    currentUser : 'domain\\user1',
    currentLCID : '1033'
  }

} ();

Now as soon as DOM is loaded, you have your local variables present that you can use right away in your control further down in the DOM (or controls etc) without waiting for any other JavaScript file to load like following:


    var currentRootWebUrl = Local.Variables.currentRootWebUrl;
    var currentWebUrl = Local.Variables.currentWebUrl;

AllowEveryoneViewItems - Enable anonymous access to files in list

Wed, 12 Dec 2012 00:00:00 GMT

SharePoint provides anonymous access to sites and document libraries etc but for that you have to set up the policy at web application level and then for specific site or document library as per your needs. But at times you don't want to allow anonymous access for entire web application or site collection but simply want to provide access to files within a specific document library without going through the trouble of configuring and managing permissions.

If you want to access documents within a document library or files in a list (as an attachment) there is one quick and easy way to do that.

There is a list property called AllowEveryoneViewItems that makes all documents within a document library anonymously accessible without dealing with permissions. There is no option in UI to set this up so you have to either do it through code, declaratively (list definition element.xml) or Powershell (depending on your situation and need).

Here is the PowerShell code for setting this property on any list or library


$web = Get-SPWeb -Identity "[site url]"
$list = $web.Lists.TryGetList("[list title]");
$list.AllowEveryoneViewItems = $true
$list.Update()

This is how you define in list definition xml:


<!--?xml version="1.0" encoding="utf-8"?-->

	<ListTemplate
		Name="[list name]"
		DisplayName="[list display name]"
		Description="[list description]"
		Type="10000"
		BaseType="0"
		Default="True"
		VersioningEnabled="TRUE"
		Hidden="FALSE"
                AllowEveryoneViewItems="TRUE"
		HiddenList="FALSE" />

And this is how you can do it using code


using (SPSite site = new SPSite("[site collection url]"))
{
   using (SPWeb web = site.OpenWeb())
   {
       SPList list = rootWeb.Lists.TryGetList("[list title]");
       if (list != null)
       {
          if (!list.AllowEveryoneViewItems)
          {
             list.AllowEveryoneViewItems = true;
             list.Update();
          }
       }
    }
}

It is important to note here that this property does not apply to all list items, but only to documents in document libraries or to attachments in list items. So you have to access this file directly for it to work ! (reference: SPList.AllowEveryoneViewItems property)

Here is the code snippet for accessing the file directly using HttpWebRequest


var fileContent = String.Empty;
//construct path to the navigation file
string fileUrl = "[full url of the file]";

//create webrequest to fetch this file!
var httpRequest = (HttpWebRequest)WebRequest.Create(fileUrl);
using (var webResponse = (HttpWebResponse)httpRequest.GetResponse())
{
   if (webResponse.StatusCode == HttpStatusCode.OK)
   {
      using (var responseStream = new StreamReader(webResponse.GetResponseStream()))
      {
          //if we get the response, read the content from file as string
          fileContent = responseStream.ReadToEnd();
      }
   }
}