Sunday, April 23, 2017

How to display Sharegate administrative dynamic reports via Power BI on SharePoint Online

After working in the SharePoint domain for a couple of years as SharePoint consultant , I am fortunate enough to help customers from many verticals such as education, utilities, banking, and Government sectors. 

A common thing which showed me why SharePoint projects (in general ,all the non-projects) get delayed are as follows: 

  • No proper requirement analysis from the key stakeholders as a partner. When project starts and new requirements come within the project which were not factored in the initial project plan. This resulted in change requests and depending on the change the timeline gets shifted.
  • No foresight in the SharePoint’s capacity planning in terms of SharePoint On-Premises environment. Please read my earlier blog about why SharePoint is slow for more details.  
  • None or very limited SharePoint Governance.
It’s not a new buzzword but SharePoint governance has become far more critical in SharePoint Online environment.   

In most of organization’s I have had worked, they have a little or no governance at all. SharePoint Governance Plan is most the important success factor.

When a SharePoint platform is given to end users, it starts out easy. However, they keep adding more content with no information architecture in place, it gets complicated. The concepts of term store, management metadata services, search service, user profile service could be new for end users and educating them is important.

In a nutshell, SharePoint Governance includes:

  • Information Management (Site planning, Navigation, Management metadata, search etc.
  •  Application Management (Customization policy, Solutions, 3rd Party)
  • Communication and training (communication plan,  training plan, support plan)
  • Roles and Responsibilities (Team definition and responsibilities)

Please refer to this excellent blog about SharePoint Governance.  
Without governance it would be like a city without any authority roles and guideline as illustrated below: 

Figure 1 Courtesy: Sharegate

To put the context with perspective in relation of the SharePoint governance, I had to worked for a client who had absolutely zero SharePoint governance. They asked us to display site collection quota on one of the site collections and a report in excel of folder sizes to end users every month end so that end users can clean the unnecessary files on their SharePoint 2013 On-Premises environment.

I created Server Object model to achieve this in my blog.
Furthermore, we had to give site access report via PowerShell who accessed the SharePoint site monthly in manual way.  

Now, if they had 30 site collections and number of folders of each site collection the effort would be much more.

In the SharePoint Online context, it would be much more complicated as we have any access to SharePoint app model to manipulate the folder size using CSOM and REST.

Sharegate is one of the best tools that I have come across for this very purpose. It has in built reports permissions report which graphically shows which use has what permissions matrix for sites (or OneDrive), site collection with basic details, external users and many more.

In new version of Sharegate, you can easily schedule tasks every day, every week, every month etc. which makes SharePoint administrator’s job much more simpler without PowerShell.

In this blog post, I will detail steps to generate SharePoint administrative reports using Sharegate and display this report on the SharePoint Online site.

Step1: Open Sharegate desktop client (version, navigate to reporting section and click on Site with basic details as shown: 

Step2: Select the sites to be included in the report and click on “Run” button.

Step3: Verify the results and click on Export button.  

Step4: Save the file to the desktop.

Step5:  Microsoft has given us a FREE business intelligence tool called Power BI. Download to your desktop as client. 

Step6: From Home tab, click Get Data and locate Excel option. 

Step7: In the Navigator, the results page has to be adjusted for Size and Date. Click on the Edit button.    

Step8: In the Title column, remove blank rows, in the Size column, click on the “Replace Values” option and Replace value MB to blank as shown:

Step9: Do the same process for the “Last Modified Date”, select the Data Type, Date field. 

Step10: Click on the Close and Apply

The Power BI report is saved as .pbix format


Now, on right side of Power BI client, we can see Visualizations and Fields that we selected in above steps. Select the Multi-row card visualization pane

Results pane, select the Title as shown:

Step12: Click on the Pie Chart in Visualization pane 

Step13: Select Title field from Results pane and drag to Details in the Visualization pane as shown: 

Step14: Select Size field from Results pane and drag to Values in the Visualization pane as shown: 

Step15: We get a dynamic SharePoint report for the size for ALL the site collections we choose from Power BI. 

We can change the view to display from information from the graph as follows:

Furthermore, we can also drill down to each site and view the details: 

Publish this report to SharePoint Online.

Step16: We need to active Power BI app on SharePoint Online. Follows these steps:

Sign up to leverage Power BI 

After few seconds, we have Power BI app activated in our SharePoint Online environment.

Go back to locate the file that we just created in Power BI a few minutes back.

and our report available on Power BI at SharePoint Online. 

Step17: File >> Publish to Web

Next, get the “Create embed code


Click in “Publish” button.

Embedded code is available in Power BI

Step18: On SharePoint Online Site, Edit the Wiki page, and insert the Embed code.

A preview will be displayed. 

Close and Save the page. Our report is now available. 

I hope this blog post helped you and please comment if it is useful.


Saturday, February 4, 2017

How I passed Nintex Workflow Admin Certificate

Hi all,

Finally, I passed my Nintex Workflow Admin Certificate and scored 251 on 04 Feb 2017. I started learning Nintex products last Nov 2016 and started preparing for last week of Dec 2016. I tried my first attempt and was unsuccessful with score of 138 out of 300 in Jan 2017. The passing score is 180.

I asked a lot of questions from Nintex community about how to pass this exam. Erick Weitkamp (from Nintex) suggested to take the full exam, sense the kind of questions and then retake it again. I followed his instructions. I am working on SharePoint online and number of questions are not exactly the exam is looking for. However, after studying for exam I am able understand the basic concepts such as LazyApproval, Installation of Nintex forms and workflow, NW Admin tool concepts, backup and restore etc.

If you do not have Nintex subscription, then you can download a free app on SharePoint online  and create a free Nintex Account and practice with exam.

Some Nintex Admin Certificate takeaways:

·         Exam is for 60 minutes and 30 questions are asked.
·         Most of topics are covered in exam syllabus. 4-5 questions are not covered though. You can perform a google search to find out answers. But take note of time. In my first attempt I was short of time.
·         Understand the LazyApproval, NW Admin tool, Installation of Nintex products, Performance guide, and Database design thoroughly.   
·         I made my own notes and checked all the questions that I could not answer in my first attempt. It really worked.
·         At this point of time, there are no videos dedicated for preparing for Nintex Workflow Admin certification. May be Nintex team can provide them in future. 

Now, time to study for Nintex Workflow Pro certificate.

Wednesday, January 11, 2017

SharePoint Online Sandbox solutions COMPLETELY Removed

Hi All, 

On 29 July 2016,  SharePoint team has completely removed the sandbox solutions from SharePoint Online.

SharePoint team is promoting Add-In model to create custom solutions.  
If you have custom solutions on SharePoint Online, they may not work.


Wednesday, December 7, 2016

How to: Change team site template to developer site. Sideloading of apps is not enabled on this site

Hi all,

While working on SharePoint Online apps via Visual Studio 2015, you may encounter some issue I faced initially.

1.    Build a simple such as Hello World, if your site is Team Site, you will get an error in  Visual Studio 2015 when deploy your app as below. Sideloading of apps is not enabled on this site

2.    After some research  I found I need to download the SharePoint Online Management Shell.

I checked all the SharePoint Online commands

3.    I directly typed the following command:
Enable-SPFeature e374875e-06b6-11e0-b0fa-57f5dfd72085 –url

and received a welcome error: ;). Basically its feature is SharePoint On-Premises environment and not for SharePoint Online.

4.    My research continued and found a script from renowned MVP Colin Phillips and download this script on my laptop.

5.    I again got an error and his script has some extra special characters that PowerShell could not recognise. So I removed all the  extra special characters and ran again.

I again got an error as follows:

6.    I ran the command with unrestricted execution policy and finally converted my team site to developer site.

I hope it helps. This is the corrected script.


Tuesday, September 20, 2016

Why SharePoint is slow?

As a SharePoint consultant, I have seen quite a few end user complaints that SharePoint is slow in different organizations. The only thing which comes for those slow environments is poor planning and design flaw of overall SharePoint architecture.

In that context, getting proper requirement analysis from the stakeholders and scope for that project extremely crucial to have a successful SharePoint implementation and end users happy.  In my 7 years of SharePoint administration and designing of SharePoint architecture, planning of SharePoint is the key and concrete scope with complete user requirements.

This blog post is devoted to this and what has to be watched out primary bottlenecks along with tools.
Let’s get started:

1.       Planning for SharePoint Infrastructure - Hardware Specifications: Check out SharePoint 2013 hardware/software requirements.    

Proper planning for hardware and supported infrastructure need to carefully done. SharePoint 2013 server RAM hungry. Therefore, many of SharePoint performance issue are attributed to lack of resources.

a.       Capacity Planning:

Is SharePoint Infrastructure designed according to Microsoft Hardware and Software minimum requirements? This is one of the most important aspect where SharePoint architect needs to talk to the infrastructure team.

Key design for SharePoint architecture includes:

i) Latency: time between a user clicks on browser and reposed back. This is something that end user complain about SharePoint being slow.

>> IE Developer tools to check the response. (It’s extremely useful )
>> Fiddler

ii) Throughput: Number of requests that as server farm can process in a unit of time. Some common examples of low throughput conditions,
processors on the farm are not over-utilized. For e.g. if CPU usage during peak hours or load spikes consistently exceeds 80 %, add more servers or redistribute services to the other farm servers, is sufficient memory on application and web servers to contain the complete cache, ensure that database servers are free of bottlenecks.

 >> Enable SharePoint Developer Dashboard.
>> SQL Server Logging Database

 Below capacity planning is specifically for SharePoint 2013.          

b.      Choosing Processors:

SharePoint Server 2013 is available only for 64-bit processors. Therefore more processors will enable us to serve greater demand.

•             WFEs: In individual web servers will scale up as we add more cores.
•             Application server:  Application server processor capacity requirements differ depending on the role of the server and services it is running.  

c.       Choosing Memory:

·         Web ServersMicrosoft recommends allocate at least 8 GB RAM on each web server and 16 GB for servers that have greater traffic or deployments with multiple application pools set up for isolation

·         Application serverApplication server processor capacity requirements differ depending on the role of the server and services it is running Ideal application server should have minimum 4 cores (quad-cores) and recommended 8 cores (octa-cores)

·         The memory requirement are of database servers are tightly dependent on the database size.

d.      Choosing Network:

There is significant traffic in a farm across the web server tier, the application server tier and the database tier. The network can easily become a bottleneck under certain conditions like dealing with very large files or very high loads.

Web Servers and application servers should be configured to use at least two network interface cards (NICs): one NIC to handle end-user traffic and other for to handle inter-server communication.

e.       Choosing Disk and Storage:

We need to ensure that at least 30% additional capacity on each disk, above our highest data estimate, to leave room for future growth.

The web and application servers have storage requirements also. In most of the production environment, Microsoft recommends that we allocate at least 200 GB disk space for OS and temp and 150GB for disk space for logs.

2.       Optimize SQL Server for SharePoint:

The database is a common area where performance hits are most prevalent. SharePoint is database intensive, about 95% of SharePoint is stored in a SQL database: files, images, videos, pages, content, user profiles, etc. It’s important to have a happy and health SQL database running.

a.       Tasks Manager on SQL Server:  Check the services which are consuming too much RAM and CPU.  We also need to configure I/O counters as well.

b.      Configure Model System Database Settings
                                                               i.      Increase auto growth settings (Use MB and not %)
                                                             ii.      Recovery model should Full.

c.       Configure TempDB System Database settings
                                                               i.      Increase auto growth settings (Use MB not %)
                                                             ii.      Recovery model should be Simple

d.      Max Degree of Parallelism
Set max degree of parallelism to 1 for instances of SQL Server that host SharePoint Databases, and it is to ensure that a single SQL Server process serves each request.

e.       Minimum Server Memory
Default is 2 Billon way too much for SQL Server. Other applications may be slow because SQL Server is consuming maximum RAM available.   

The following table gives RAM usage for SQL Server 2014

Server memory(GB)
SQL Server Max Memory

3.       Optimize SharePoint:

Above points are primarily dictates (Hardware Requirements and Capacity Planning) are fundamental concepts and designed by SharePoint architect gather user requirements to provide a pilot SharePoint environment. Generally speaking, while setting SharePoint infrastructure & SQL server is quite easy to set up. The primary challenge is when more data is added to SharePoint lists/libraries and users keep complaining that SharePoint has become slow. The below points slow SharePoint:

a.       Tasks Manager on servers:  Check the services which are consuming too much RAM and CPU. Use PowerShell script the services which are consuming more RAM.

b.      Offload the SharePoint logs from C drive to other some drive on server.
SharePoints logs are usually on the drive it was installed on, and in most cases that’s the C drive of the server. It’s recommended to move these logs to another partition, and preferably different drives altogether.

                                                              i.      Go to Central Administration >> Monitoring >> Configure Diagnostic Logging.
                                                             ii.      Note the top part, Event Throttling. This specifies what SharePoint will log. If any of these have been changed from default, they’ll be in bold. Your ideal settings for all of them are
                                                           iii.      Least critical event to report to the event log: Information
                                                           iv.      Least critical event to report to the trace log: Medium
                                                             v.      In the next section, Event Log Flood Protection, make sure that’s checked.
                                                           vi.      And in Trace Log section, feel free to move the logs to a different drive to either free up space on the current drive or to a unique drive. You may limit the amount of days or space the logs use per your preference.
                                                          vii.      Click OK.

c.       Web Parts:

                                                              i.      Large number of closed Web Parts, often on the home pages, will slow down a page. Every closed Web Part causes a small amount of processing overhead, and it can add up. To check this, add ?contents=1 to the page’s URL and remove any Web Parts which aren’t displayed. It can make a huge difference.

                                                             ii.      Enable Developer Dashboard - Any custom developed web part or third party web pat that is slowing the page. Developer dashboard contains information about Web Parts, events, DB calls and a whole lot of nifty information.

d.      Large lists and libraries containing several thousand items can slow things down a lot too.
Optimize: Content Query Web Part, list view and metadata navigation and search

e.       Consider Warming up your SharePoint Site
Warming up the pages across your SharePoint site is a good idea for two reasons. Firstly, ASP.NET’s Just-In-Time compiler will often cause the first access of a site to be extremely slow compared to usual. Secondly, the caching techniques discussed previously only kick in once a page has been visited for the first time. By running a script or job to warm up the site you can avoid the ‘first hit’ performance lag associated with these issues. There are a number of sources of information regarding this topic but one such article that seems to collate a lot of them is Wahid Saleemi’s Roundup: SharePoint Warm-Up Scripts.

f.        Make Use of SharePoint’s Built-In Caching Functionality

                                                               i.      Output Caching (Configurable)
The concept of Output Caching is something that natively comes with SharePoint 2010, as it builds on and relies on ASP.NET caching techniques. This means that you can configure your SharePoint 2010 site to cache the Pages it outputs. The reasoning behind caching a page is obviously that it takes time to generate the content on any page, and on a heavily accessed site it would be a performance impact to generate a new page on every request – that’s where Output Caching comes in handy.

                                                             ii.      Object Caching (Configurable)
In SharePoint Server 2010 you’ve got the option to use Object Caching as well, which is a mechanism to cache specific page items. This is especially likable if you’re playing around with Cross-List data queries and need to cache the results of such a query.

                                                           iii.      BLOB Cache (Configurable)
In SharePoint 2010 you also have something called the BLOB Cache, which is a disk-based caching mechanism that caches resources on the disk. Normally these resources are files served by a web page and are named Binary Large OBjects (BLOB).

    enabled="false" />
Make Use of SharePoint’s Built-In Caching Functionality

g.       Distributed Cache Service Application:

SharePoint Distributed cache service is an automatic service.  However, if we configure to a dedicated distributed cache service you may face some challenge.

I had earlier written a detailed blog specifically the challenges with Distributed Cache service.   

I had a chance to work an outsourced (inexperienced) vendor and they claimed that SharePoint server on pre-release were slow servers. As a matter of fact, vendors has absolutely no idea to solve the issue. It is a migration project from SharePoint 2010 to SharePoint 2013 and the SharePoint 2010 had a heavily custom site and branding.

This how I found the root cause.

                                                               i.      Hardware specifications:  I checked both the staging environment and pre-release environment hardware specs. There were absolutely identical. So it's ruled out.

                                                             ii.      SQL Server: SQL server settings, parameters and assigned memory were identical. So it's ruled out.

                                                           iii.      Firewall: The Firewall rules were verified for both the environments there were identical. So it's ruled out.

                                                           iv.      Search settings: Content Sources were configured as incremental. Continuous crawl was configured. So it's ruled out.

                                                             v.      Distributed cache settings: Service is started status on App server. It's not enabled on WEB server.

                                                           vi.      Developer Dashboard: I turned on Developer dashboard both the staging and pre-release environments to check the page performance. I noticed staging environment Internet Explorer developer tools showed page load 1.5 seconds but whereas pre-release environment showed 6 seconds for the custom branding site.

I checked if any error ULS tab for developer dashboard. I couldn't find any error that showed whatsoever that jump of execution that 6 seconds.
ULS viewer didn't give error that gives any indication of 6 sec jump. However, I know it could be of Distributed Cache service. What I know is that there is some wrong in this pre-release environment.

                                                          vii.      New Web Application: To narrow down this issue, I created a Web application using team template. I asked vendor if they have code related My Site or Distributed cache. They do have some custom code related to My Site. I removed their solution.

 Turn off and turn on the Distributed cache sever. Restarted the timer service on App and web server. The new Web application's site loads within 1 second.
It tuned out that issue was custom code made web application non performant and vendor admitted that it was their code that had an issue.

h.      Search Service Application:

One of the sluggish performance could from Search service application. Following the tips to improve:

                                                               i.      Ensure that we have SharePoint 2013 March patch.
                                                             ii.      Reduce the Search crawl time interval.
                                                           iii.      Major change could be change the Search’s server’s noderunner. Make a copy of noderunner.exe.config (C:\Program Files\Microsoft Office Servers\15.0\Search\Runtime\1.0\noderunner.exe.config)

The default is Maximum. We can adjust the search performance by setting PartlyReduced or Reduced

Set-SPEnterpriseSearchService -PerformanceLevel Reduced

I hope this blog helps and please comment if this blog post is helpful.


·         Planning Distributed cache in SharePoint 2013.
·         Optimize Search in SharePoint 2013., Fix Search Performance   
·         Warmup SharePoint 2013 sites

·         10 Reasons Why SharePoint Performance Can Slow-

·         SharePoint Running Slow? Here’s How We Solved the Mystery!

·         My Users Don’t Like SharePoint because it is too slow!

·         SharePoint site are slow intermittently: SharePoint 2010. -

·         Intermittent Database Server Connectivity and Microsoft SharePoint Products and Technologies           

How to display Sharegate administrative dynamic reports via Power BI on SharePoint Online

After working in the SharePoint domain for a couple of years as SharePoint consultant , I am fortunate enough to help customers from man...